In fact for this price the camera is quite nice. It has a solid aluminium outdoor case, the picture quality is great (even in night) because of the Sony Exmor CMOS; it has the ONVIF. So overall this is a nice piece of hardware. Only the software is not ideal (but when talking into account the price, it is understandable).
The camera has the following web interface:
The interface doesn’t allow to setup all camera parameters, so additional windows software was needed to configure it. I was using the “Golbong Software” to configure services, like NTP.
Last Tuesday after the camera maintenance reboot (oh yes: the camera is rebooting itself every week by default! As a programmer I hate this kind of workarounds. Why not making software reliable and got rid of all memory leaks?), the camera suddenly stopped working (and also doesn’t respond to ping). I thought that it may be a one-time problem, so I just simply cycled power, and it was working again.
Unfortunately the same problem occurred again in about 10 hours. I was able to see this in grafana:
The first red line is the “maintenance reboot”, then the pings are raising and it freezes, the same problem occurred again in similar time.
Generally the camera was working properly for about 10-11 hrs from startup/reboot.
Getting shell 🙂
I was started to googling information about the camera software. Based on the version number, I was able to find two interested articles about similar cameras:
I found out that the camera has a “hidden” telnet debug service on TCP port 9527. After connecting to this port I’ve got a plenty of “garbage” on it:
… but at the end it was asking me for username and password. After providing the user and password (the same as for the web interface) I was able to do some nice things. There is a help command which shows available commands. Of course for me the most interesting was the “shell” command 🙂
I typed it and got the shell, but there was some problem with EOL characters, so I followed the guide on some of the articles and typed:
This opened a regular telnet on port 23 on the device. So from another console I was able to telnet into it. Marcus Jenkins cracked a root password (xmhdipc) so logging to this device was really simple:
So I was able to monitor processes, free memory and connections. I was waiting for the next time when the camera again raises the ICMP responses, which clearly shows that it will freeze soon. When it happened I found out that killing the “Sofia” process makes the ping lowered back to normal (BTW: The debug messages which are in the debug port 9527 are from this Sofia process).
Suddenly the camera has rebooted automatically and I’ve been disconnected. It was probably some watchdog. But I just know that the Sofia process has to be responsible for the problem.
I also discovered the free memory before was 14804 and when the problem was occurring it was 11116. So for the first sight it looks like a memory leak or something similar to me.
Next time I located the problem. The camera was connecting to some cloud services (vMEyeCloud) to china. And was also trying to obtain the public IP address via some service. It seems that last days the server response has to be changed somehow because I can see that I have a plenty of CLOSE_WAIT connections which was raising:
# netstat -nap|grep CLOSE_WAIT tcp 2 0 192.168.0.12:41399 220.127.116.11:8086 CLOSE_WAIT 742/Sofia tcp 2 0 192.168.0.12:41405 18.104.22.168:8086 CLOSE_WAIT 742/Sofia tcp 2 0 192.168.0.12:41401 22.214.171.124:8086 CLOSE_WAIT 742/Sofia tcp 2 0 192.168.0.12:41403 126.96.36.199:8086 CLOSE_WAIT 742/Sofia tcp 2 0 192.168.0.12:41398 188.8.131.52:8086 CLOSE_WAIT 742/Sofia tcp 2 0 192.168.0.12:41406 184.108.40.206:8086 CLOSE_WAIT 742/Sofia tcp 2 0 192.168.0.12:41397 220.127.116.11:8086 CLOSE_WAIT 742/Sofia tcp 2 0 192.168.0.12:41404 18.104.22.168:8086 CLOSE_WAIT 742/Sofia tcp 2 0 192.168.0.12:41400 22.214.171.124:8086 CLOSE_WAIT 742/Sofia [...]
During the time when it was raising the ping responses, I had 457 such entries. Maybe not so big number, but it was enough for this SOC.
The first thing which comes to my mind was to try to block the internet for this camera. So I did this … and guess what? No more raising connection count.
After waiting reasonable time I am now sure that it was the problem! The camera is not freezing anymore 🙂
Probably the cloud services are also not working, but I am not using it. Besides it is now configured without internet access, which has to be done like this from beginning. Why it has to “spy” and connect to foreign servers? 🙂