[Solved] After updates, desktop locks frequently (nvidia)

This forum is dedicated to advanced help and support :

Ask here your questions about advanced usage of Mageia. For example you may post here all your questions about network and automated installs, complex server configurations, kernel tuning, creating your own Mageia mirrors, and all tasks likely to be touchy even for skilled users.

[Solved] After updates, desktop locks frequently (nvidia)

Postby kbulgrien » Mar 16th, '20, 05:40

I started getting many desktop lockups a day after a recent round of updates. I am fully current, and depend on the NVidia DKMS drivers. When the system is completely unresponsive, I can SSH into the system, and from a command-line everything seems to work fine, but I don't know how to recover the desktop, and end up having to `sudo reboot`. The desktop is usually fully locked, but sometimes a mouse cursor is visible and moveable though ineffective. Ctrl-Alt-Backspace and any other keyboard combination, even Ctrl-Alt-Del are ineffective. Trying keys like this inevitably disappear the cursor. Reboot from SSH session shows nothing until the BIOS boot splash comes up. I can sometimes tell when this is going to happen as often window frames start getting glitchy (black/white zebra stripes), or bits of the screen start scrambling (blocky colors blinking), or maybe some areas of a which show fully scrambled pixels. It will often run fine for a while, but once this starts to happen, it will lock up. The system is otherwise identically configured to the way it was before the updates when it was rock solid and never locked up. I have a GeForce GTX 750 Ti and depend on the DKMS Nvidia drivers as I have never been able to do what I need to do with the Nouveau drivers. All this appears to have started when I noticed DKMS trying to rebuild new drivers every boot. NVIDIA drivers broken by updates per my experience seem to be a recurring theme. At some point, the drivers stopped rebuilding at boot, but I'm still getting these hard lockups. I don't really know how to troubleshoot this. I am generally gaming when this happens. I don't really heavily use the system otherwise. Usually I notice the glitching in the game first, but the window corruption also is seen on some other application windows too. I'm using plasmashell. top results while the GUI is locked up tight don't seem abnormal... no high CPU or RAM usage conditions. The game appears to actually be running... i.e. once I started crafting operations that would take many minutes. I left the system running while experimenting with getting in via SSH from another system. After I'd rebooted the system, and logged back into the game, it was as if the game had actually kept running normally even though the screen/keyboard/mouse were locked hard. Whatever this is, its a desktop malfunction, and not a general system lockup.

These are the RPMS recently updated by date. I can't say for sure if the lock ups started on March 12 or March 14.

Code: Select all
# Generated:      Sun 15 Mar 2020 10:31:46 PM CDT
# Installed RPMS: 2283
2020/03/14/16:04:54 kernel-userspace-headers       5.5.9-1.mga7
2020/03/14/16:04:48 kernel-desktop-5.5.9-1.mga7    1-1.mga7
2020/03/14/16:04:42 kernel-desktop-latest          5.5.9-1.mga7
2020/03/14/16:04:41 cpupower                       5.5.9-1.mga7
2020/03/14/16:04:40 firefox                        68.6.0-1.mga7
2020/03/14/16:04:31 firefox-en_US                  68.6.0-1.mga7
2020/03/14/16:04:29 lib64nss3                      3.51.0-1.mga7
2020/03/14/16:04:27 nss                            3.51.0-1.mga7
2020/03/14/16:04:22 kernel-desktop-devel-5.5.9-1.mga7 1-1.mga7
2020/03/14/16:04:19 kernel-desktop-devel-latest    5.5.9-1.mga7
2020/03/12/17:46:52 sane-backends-iscan            1.0.27-4.mga7
2020/03/12/11:39:47 libmesagl1                     20.0.1-1.mga7
2020/03/12/11:39:46 libmesavulkan-drivers          20.0.1-1.mga7
2020/03/12/11:39:45 libgio2.0_0                    2.60.2-1.3.mga7
2020/03/12/11:39:44 libglib2.0_0                   2.60.2-1.3.mga7
2020/03/12/11:39:43 lib64xatracker2                20.0.1-1.mga7
2020/03/12/11:39:43 lib64seccomp2                  2.4.2-1.mga7
2020/03/12/11:39:42 lib64mesavulkan-drivers        20.0.1-1.mga7
2020/03/12/11:39:40 lib64osmesa8                   20.0.1-1.mga7
2020/03/12/11:39:39 libpcre1                       8.44-1.mga7
2020/03/12/11:37:36 libdri-drivers                 20.0.1-1.mga7
2020/03/12/11:37:31 libgomp1                       8.4.0-1.mga7
2020/03/12/11:37:30 libgfortran5                   8.4.0-1.mga7
2020/03/12/11:37:30 lib64ilmbase24                 2.3.0-1.1.mga7
2020/03/12/11:37:29 gcc-plugins                    8.4.0-1.mga7
2020/03/12/11:37:28 lib64http-parser2              2.9.3-1.mga7
2020/03/12/11:37:27 x11-server-xwayland            1.20.7-1.mga7
2020/03/12/11:37:26 x11-server-xorg                1.20.7-1.mga7
2020/03/12/11:37:24 lib64epoxy-devel               1.5.4-1.mga7
2020/03/12/11:37:22 gcc-c++                        8.4.0-1.mga7
2020/03/12/11:37:20 lib64icu-devel                 63.1-1.2.mga7
2020/03/12/11:37:17 libglapi0                      20.0.1-1.mga7
2020/03/12/11:37:16 libquadmath0                   8.4.0-1.mga7
2020/03/12/11:37:14 lib64icu63                     63.1-1.2.mga7
2020/03/12/11:37:07 kernel-desktop-devel-5.5.6-2.mga7 1-1.mga7
2020/03/12/11:37:03 gcc                            8.4.0-1.mga7
2020/03/12/11:37:00 binutils                       2.33.1-1.mga7
2020/03/12/11:36:59 lib64mesagl1-devel             20.0.1-1.mga7
2020/03/12/11:36:58 x11-server-common              1.20.7-1.mga7
2020/03/12/11:36:57 lib64mesagl1                   20.0.1-1.mga7
2020/03/12/11:36:56 lib64dri-drivers               20.0.1-1.mga7
2020/03/12/11:36:53 mesa                           20.0.1-1.mga7
2020/03/12/11:36:52 libstdc++-devel                8.4.0-1.mga7
2020/03/12/11:36:51 libstdc++6                     8.4.0-1.mga7
2020/03/12/11:36:50 libgcc1                        8.4.0-1.mga7
2020/03/12/11:36:48 rsync                          3.1.3-4.mga7
2020/03/12/11:36:48 lib64gd3                       2.2.5-5.2.mga7
2020/03/12/11:36:47 lib64solv1                     0.7.4-1.1.mga7
2020/03/12/11:36:46 lib64pq5                       11.7-2.mga7
2020/03/12/11:36:46 lib64archive13                 3.4.0-1.1.mga7
2020/03/12/11:36:44 lib64glib2.0-devel             2.60.2-1.3.mga7
2020/03/12/11:36:43 gcc-cpp                        8.4.0-1.mga7
2020/03/12/11:36:42 icu63-data                     63.1-1.2.mga7
2020/03/12/11:36:38 kernel-desktop-5.5.6-2.mga7    1-1.mga7
2020/03/12/11:36:32 lib64pcre-devel                8.44-1.mga7
2020/03/12/11:36:31 lib64pcre32_0                  8.44-1.mga7
2020/03/12/11:36:31 lib64pcre16_0                  8.44-1.mga7
2020/03/12/11:36:31 glib-gettextize                2.60.2-1.3.mga7
2020/03/12/11:36:30 lib64mesaegl1-devel            20.0.1-1.mga7
2020/03/12/11:36:30 lib64gbm1-devel                20.0.1-1.mga7
2020/03/12/11:36:29 glib2.0-common                 2.60.2-1.3.mga7
2020/03/12/11:36:28 lib64mesakhr-devel             20.0.1-1.mga7
2020/03/12/11:36:27 lib64glib2.0_0                 2.60.2-1.3.mga7
2020/03/12/11:36:27 lib64gio2.0_0                  2.60.2-1.3.mga7
2020/03/12/11:36:26 lib64pcre1                     8.44-1.mga7
2020/03/12/11:36:26 lib64mesaegl1                  20.0.1-1.mga7
2020/03/12/11:36:26 lib64glapi0-devel              20.0.1-1.mga7
2020/03/12/11:36:25 lib64epoxy0                    1.5.4-1.mga7
2020/03/12/11:36:24 lib64glapi0                    20.0.1-1.mga7
2020/03/12/11:36:24 lib64gbm1                      20.0.1-1.mga7
2020/03/12/11:34:18 glibc-devel                    2.29-20.mga7
2020/03/12/11:34:13 glibc                          2.29-20.mga7
Attachments
pre-lockup_screenshot.png
Attached is a screenshot of some of the GUI artifacts that let me know something is going wrong. Usually this is with the game screen, but has also been visible in other application window controls.
pre-lockup_screenshot.png (1.48 MiB) Viewed 1571 times
Last edited by kbulgrien on Mar 18th, '20, 07:11, edited 2 times in total.
kbulgrien
 
Posts: 11
Joined: Oct 10th, '11, 07:00

Re: After updates, desktop fully locks frequently (nvidia)

Postby kbulgrien » Mar 16th, '20, 06:36

Actually, I'm going to guess that the lockups ARE nouveau... It would appear that at some point the system gave up trying to recompile the DKMS module and reverted to nouveau. I re-ran the graphics driver set up and responded that I wanted the proprietary driver... now I'm seeing `ps` results with "nvidia" in them, and I see in the old X startup log that nouveau was loading:

Code: Select all
-rw-r--r--  1 root   root              40779 Mar 14 22:47 Xorg.1.log.old
-rw-r--r--  1 root   root               5720 Mar 15 23:20 Xorg.9.log
[root@master log]# grep nouveau *
Xorg.1.log.old:[  2064.201] (II) LoadModule: "nouveau"
Xorg.1.log.old:[  2064.202] (II) Loading /usr/lib64/xorg/modules/drivers/nouveau_drv.so
Xorg.1.log.old:[  2064.238] (II) Module nouveau: vendor="X.Org Foundation"
Xorg.1.log.old:[  2064.239] (II) [drm] nouveau interface version: 1.3.1
Xorg.1.log.old:[  2064.421] (II) NOUVEAU(0): [DRI2]   DRI driver: nouveau
Xorg.1.log.old:[  2064.421] (II) NOUVEAU(0): [DRI2]   VDPAU driver: nouveau
Xorg.1.log.old:[  2064.482] (II) AIGLX: Loaded and initialized nouveau
Last edited by isadora on Mar 16th, '20, 09:07, edited 1 time in total.
Reason: Placed command-output in between [CODE]-tags for better readability ;)
kbulgrien
 
Posts: 11
Joined: Oct 10th, '11, 07:00

Re: After updates, desktop fully locks frequently (nvidia)

Postby doktor5000 » Mar 16th, '20, 19:25

If at the most recent boot the proprietary driver was loaded, then it will be used until you reboot or until you shutdown and reconfigure the X server. It will not suddenly change to nouveau during runtime just because an update ran - just for clarification.

For how to restart your desktop session, this as root should work - although you lose everything unsaved inside of your desktop session, but you're probably aware
Code: Select all
systemctl restart displaymanager


For nvidia driver vs. nouveau, I've also noticed quite a few graphics corruption issues and/or hangs when coming back from suspend / hibernate, so I finally switched to nouveau because that at least works for me regarding suspend / hibernate.

You may want to post your current xorg.conf and also the output as root of
Code: Select all
dkms status
rpm -qa|grep -iE "kernel|nvidia|dkms"
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17659
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: After updates, desktop fully locks frequently (nvidia)

Postby kbulgrien » Mar 16th, '20, 21:20

It was not intended to imply that the driver change occurred at run-time. The driver did, however, apparently revert to nouveau without user direction. Reboots occurred during the time when the system became unstable.

Thank-you for the hint regarding systemctl restart displaymanager. I had not yet acclimated to this systemd change. I tried systemctl restart dm and some other things, but have not succeeded at locating displaymanager with man systemctl or systemctl --help hints. I wonder what which command would have shown that displaymanager was an option? Perhaps I can remember this as I've set my command-line history to be very long.

Also, I failed to mention that during the period of instability, the pre-desktop black screen had two nouveau-related error messages, but I have not found them in log files after the fact. I believe they were something like what follows except there were two of them (likely because I run a dual monitor setup):

Code: Select all
nouveau 0000:01:00.0: bus: MMIO read of 00000000 FAULT at ...


As to suspend / hibernate, this was not a factor, and, moreover, if their use precludes use of the proprietary drivers, then suspend / hibernate are not viable options here; years of experience with nouveau have produced little other than trouble and pain, though at times it has been necessary to use nouveau when the proprietary drivers were broken for one reason or another.

Code: Select all
$ dkms status
nvidia-current, 430.64-4.mga7.nonfree, 5.5.9-desktop-1.mga7, x86_64: installed

$ rpm -qa|grep -iE "kernel|nvidia|dkms"kernel-desktop-devel-5.5.9-1.mga7-1-1.mga7
kernel-desktop-devel-5.5.6-2.mga7-1-1.mga7
kernel-desktop-latest-5.5.9-1.mga7
nvidia-current-utils-430.64-4.mga7.nonfree
dkms-2.0.19-40.mga7
kernel-desktop-5.5.6-2.mga7-1-1.mga7
x11-driver-video-nvidia-current-430.64-4.mga7.nonfree
nvidia-current-doc-html-430.64-4.mga7.nonfree
kernel-desktop-devel-latest-5.5.9-1.mga7
kernel-desktop-5.5.9-1.mga7-1-1.mga7
dkms-minimal-2.0.19-40.mga7
kernel-userspace-headers-5.5.9-1.mga7
kernel-firmware-20190603-1.mga7
kernel-firmware-nonfree-20191220-1.mga7.nonfree
dkms-nvidia-current-430.64-4.mga7.nonfree


It is currently believed that running set up to expressly request the proprietary driver returned the system to stable operation once the system rebooted. It was possible to run uninterrupted for over an hour after making that change, but more runtime is needed to assure the system is indeed stable again.
kbulgrien
 
Posts: 11
Joined: Oct 10th, '11, 07:00

Re: After updates, desktop fully locks frequently (nvidia)

Postby kbulgrien » Mar 16th, '20, 21:47

This makes it clearer that a driver change did occur. I do not have the older stable configuration in use before the instability started.

Code: Select all
$ ls -l /etc/X11/xorg.conf*
-rw-rw---- 1 root root 2729 Mar 15 23:20 /etc/X11/xorg.conf
-rw-r--r-- 1 root root 2729 Mar 14 11:54 /etc/X11/xorg.conf.mga1584204858
-rw-r--r-- 1 root root 2730 Mar 14 11:54 /etc/X11/xorg.conf.old
-rw-rw---- 1 root root 2729 Mar 15 23:20 /etc/X11/xorg.conf.test


Code: Select all
$ diff -u /etc/X11/xorg.conf.old /etc/X11/xorg.conf
--- /etc/X11/xorg.conf.old      2020-03-14 11:54:24.534282003 -0500
+++ /etc/X11/xorg.conf  2020-03-15 23:20:17.708576084 -0500
@@ -63,10 +63,10 @@
     Identifier "device1"
     VendorName "NVIDIA Corporation"
     BoardName "NVIDIA GeForce 635 series and later"
-    Driver "nouveau"
+    Driver "nvidia"
     Option "DPMS"
-    Option "DynamicTwinView" "false"
     Option "AddARGBGLXVisuals"
+    Option "DynamicTwinView" "false"
 EndSection

 Section "Screen"


Code: Select all
$ cat /etc/X11/xorg.conf
# File generated by XFdrake (rev 262502)

# **********************************************************************
# Refer to the xorg.conf man page for details about the format of
# this file.
# **********************************************************************

Section "ServerFlags"
    Option "DontZap" "False" # disable <Ctrl><Alt><BS> (server abort)
    AllowMouseOpenFail # allows the server to start up even if the mouse does not work
    #DontZoom # disable <Ctrl><Alt><KP_+>/<KP_-> (resolution switching)
EndSection

Section "Module"
    Load "v4l" # Video for Linux
EndSection

Section "Monitor"
    Identifier "monitor1"
    VendorName "Generic"
    ModelName "Flat Panel 1920x1200"
    HorizSync 31.5-74.5
    VertRefresh 56.0-65.0

    # Apple iMac modes
    ModeLine "1024x768"   78.525 1024 1049 1145 1312   768  769  772  800 +hsync +vsync
    ModeLine "800x600"    62.357  800  821  901 1040   600  601  604  632 +hsync +vsync
    ModeLine "640x480"    49.886  640  661  725  832   480  481  484  514 +hsync +vsync

    # Apple monitors tend to do 832x624
    ModeLine "832x624"    57      832  876  940 1152   624  625  628  667 -hsync -vsync

    # Apple PowerBook G3
    ModeLine "800x600"    100     800  816  824  840   600  616  624  640 -hsync -vsync

    # Apple TI Powerbook
    ModeLine "1152x768"   78.741 1152 1173 1269 1440   768  769  772  800 +vsync +vsync

    # Pismo Firewire G3
    ModeLine "1024x768"   65     1024 1032 1176 1344   768  771  777  806 -hsync -vsync

    # iBook2
    ModeLine "1024x768"   65     1024 1048 1184 1344   768  771  777  806 -hsync -vsync

    # 17" Apple Studio Display
    ModeLine "1024x768"   112.62 1024 1076 1248 1420 768 768 780 808 +hsync +vsync

    # HiRes Apple Studio Display
    ModeLine "1280x1024"  135    1280 1288 1392 1664  1024 1027 1030 1064

    # Another variation
    ModeLine "1280x1024"  134.989 1280 1317 1429 1688  1024 1025 1028 1066 +hsync +vsync

    # TV fullscreen mode or DVD fullscreen output.
    # 768x576 @ 79 Hz, 50 kHz hsync
    ModeLine "768x576"     50.00  768  832  846 1000   576  590  595  630

    # 768x576 @ 100 Hz, 61.6 kHz hsync
    ModeLine "768x576"     63.07  768  800  960 1024   576  578  590  616
EndSection

Section "Device"
    Identifier "device1"
    VendorName "NVIDIA Corporation"
    BoardName "NVIDIA GeForce 635 series and later"
    Driver "nvidia"
    Option "DPMS"
    Option "AddARGBGLXVisuals"
    Option "DynamicTwinView" "false"
EndSection

Section "Screen"
    Identifier "screen1"
    Device "device1"
    Monitor "monitor1"
    DefaultColorDepth 24
EndSection

Section "ServerLayout"
    Identifier "layout1"
    Screen "screen1"
EndSection


Code: Select all
$ cat /etc/X11/xorg.conf.old
# File generated by XFdrake (rev 262502)

# **********************************************************************
# Refer to the xorg.conf man page for details about the format of
# this file.
# **********************************************************************

Section "ServerFlags"
    Option "DontZap" "False" # disable <Ctrl><Alt><BS> (server abort)
    AllowMouseOpenFail # allows the server to start up even if the mouse does not work
    #DontZoom # disable <Ctrl><Alt><KP_+>/<KP_-> (resolution switching)
EndSection

Section "Module"
    Load "v4l" # Video for Linux
EndSection

Section "Monitor"
    Identifier "monitor1"
    VendorName "Generic"
    ModelName "Flat Panel 1920x1200"
    HorizSync 31.5-74.5
    VertRefresh 56.0-65.0

    # Apple iMac modes
    ModeLine "1024x768"   78.525 1024 1049 1145 1312   768  769  772  800 +hsync +vsync
    ModeLine "800x600"    62.357  800  821  901 1040   600  601  604  632 +hsync +vsync
    ModeLine "640x480"    49.886  640  661  725  832   480  481  484  514 +hsync +vsync

    # Apple monitors tend to do 832x624
    ModeLine "832x624"    57      832  876  940 1152   624  625  628  667 -hsync -vsync

    # Apple PowerBook G3
    ModeLine "800x600"    100     800  816  824  840   600  616  624  640 -hsync -vsync

    # Apple TI Powerbook
    ModeLine "1152x768"   78.741 1152 1173 1269 1440   768  769  772  800 +vsync +vsync

    # Pismo Firewire G3
    ModeLine "1024x768"   65     1024 1032 1176 1344   768  771  777  806 -hsync -vsync

    # iBook2
    ModeLine "1024x768"   65     1024 1048 1184 1344   768  771  777  806 -hsync -vsync

    # 17" Apple Studio Display
    ModeLine "1024x768"   112.62 1024 1076 1248 1420 768 768 780 808 +hsync +vsync

    # HiRes Apple Studio Display
    ModeLine "1280x1024"  135    1280 1288 1392 1664  1024 1027 1030 1064

    # Another variation
    ModeLine "1280x1024"  134.989 1280 1317 1429 1688  1024 1025 1028 1066 +hsync +vsync

    # TV fullscreen mode or DVD fullscreen output.
    # 768x576 @ 79 Hz, 50 kHz hsync
    ModeLine "768x576"     50.00  768  832  846 1000   576  590  595  630

    # 768x576 @ 100 Hz, 61.6 kHz hsync
    ModeLine "768x576"     63.07  768  800  960 1024   576  578  590  616
EndSection

Section "Device"
    Identifier "device1"
    VendorName "NVIDIA Corporation"
    BoardName "NVIDIA GeForce 635 series and later"
    Driver "nouveau"
    Option "DPMS"
    Option "DynamicTwinView" "false"
    Option "AddARGBGLXVisuals"
EndSection

Section "Screen"
    Identifier "screen1"
    Device "device1"
    Monitor "monitor1"
    DefaultColorDepth 24
EndSection

Section "ServerLayout"
    Identifier "layout1"
    Screen "screen1"
EndSection
kbulgrien
 
Posts: 11
Joined: Oct 10th, '11, 07:00

Re: After updates, desktop fully locks frequently (nvidia)

Postby doktor5000 » Mar 17th, '20, 01:39

kbulgrien wrote:Thank-you for the hint regarding systemctl restart displaymanager. I had not yet acclimated to this systemd change. I tried systemctl restart dm and some other things, but have not succeeded at locating displaymanager with man systemctl or systemctl --help hints. I wonder what which command would have shown that displaymanager was an option? Perhaps I can remember this as I've set my command-line history to be very long.

systemctl or systemctl --list-units would shows you the loaded units and their state.
For a quick overview on commands I can recommend https://fedoraproject.org/wiki/SysVinit ... Cheatsheet or maybe https://access.redhat.com/articles/systemd-cheat-sheet

displaymanager is the name of such a unit, and it's not an option for systemctl, but for the systemctl restart action. You can also use regular bash completion using tab key to show the available commands and arguments.
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17659
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: After updates, desktop fully locks frequently (nvidia)

Postby kbulgrien » Mar 17th, '20, 04:33

systemctl or systemctl --list-units would shows you the loaded units and their state.


Actually, there was no displaymanager listed. Perhaps I customized my system somewhat
from a default install. Getting more creative with case-insensitive searches, I eventually found:

Code: Select all
$ systemctl list-units --all | grep -i manager
...
  sddm.service                                                                                          loaded    active   running   Simple Desktop Display Manager                         
...
kbulgrien
 
Posts: 11
Joined: Oct 10th, '11, 07:00

Re: After updates, desktop fully locks frequently (nvidia)

Postby doktor5000 » Mar 17th, '20, 13:14

kbulgrien wrote:Actually, there was no displaymanager listed.

If you look at the full details of sddm.service, you will see that display-manager is an alias name for it.

Sorry, that was my bad, typed from memory - systemctl list-unit-files will also show the alias names, e.g. display-manager. But that's not really obvious, granted.
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17659
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: [Solved] After updates, desktop locks frequently (nvidia

Postby kbulgrien » Mar 18th, '20, 07:14

Desktop lockups stopped after kicking nouveau to the curb.
kbulgrien
 
Posts: 11
Joined: Oct 10th, '11, 07:00


Return to Advanced support

Who is online

Users browsing this forum: No registered users and 1 guest

cron