Hacking the Paralenz Vaquita (2nd Generation) dive camera
Background
Paralenz was formed in 2015, in part because the existing action camera market suffered from limitations when used underwater. In an effort to produce a diving camera “by divers, for divers”, Paralenz came up with their first Dive Camera, followed soon after by the Dive Camera+ which had a number of unique features, such as digital depth compensation for loss of red light, as well as dive data logging which could be automatically overlayed on recorded video.
Following on from the success of the Dive Camera and Dive Camera+, Paralenz launched the Vaquita in April 2021, followed soon after by a revised model, the Vaquita 2nd Generation in August 2022, again iterating on and improving the features from the earlier models.
Unfortunately things apparently did not go well for the company, which subsequently filed for bankruptcy a mere two months after the launch of the Vaquita 2nd Generation camera in October 2022.
As the company was declared bankrupt and subsequently went into liquidation, all of its online presence went dark, making it nearly impossible to get firmware updates and causing the Paralenz apps on Android and iOS to lose a significant portion of their functionality.
Frustratingly, this has meant it is now nearly impossible to gain access to any agreements and usage licenses, apart from those in the Paralenz app, which seem to be specific to the app alone. With little else to work with, I decided it was time to start investigating what lies under the hood of my Vaquita 2nd Generation camera.
The Vaquita Diving Camera
The Vaquita 2nd Generation dive camera is a substantial improvement on the older Dive Camera+, featuring an OLED viewfinder along with higher possible video resolution and framerates as well as substantially improved DCC (Depth-controlled Colour Correction) as well as onboard GPS and a multitude of other features. Unfortunately many of the features are incomplete or lacking on the software side of things, although technically the hardware is meant to be able to support them.
To that end, I decided to see if I could get into whatever operates the camera, starting with connecting to the camera’s wifi hotspot and seeing what services are presented to connected devices.
A port scan turned up little of interest, other than an open port 21 (FTP). Connecting to the port presented a vsftpd login prompt which rejected logins, until I ended up trying the rather obvious combination of “vaquita” for both the username and password.
Suddenly, things were a LOT more interesting!
The default path was actually ‘/var/www/’, but was actually the root path of the SD card. Dropping up each level showed the usual paths seen on linux based systems, and gave me access to all sorts of useful stuff, especially inside ‘/tmp/’ where I found some log files, and inside ‘/usr/local/share/script/’ where I found a large number of shell scripts that seemed connected with the operation of the camera. Unfortunately, there did not seem to be any way to upload any files except to the SD card… I was lacking either the permission to write files in most locations, or the filesystem was mounted as read only.
Getting a useful shell
The first thing to do was to pull down any/all scripts found inside ‘/usr/local/share/script/’ which looked interesting, as well as checking the contents of ‘/etc/init.d/’, which for some reason the FTP server had access to (apparently security was not a particular concern when building this camera firmware!). Investigating the scripts folder turned up shell scripts for setting the camera into various states, such as Mass Storage mode, USB ethernet mode or even USB serial console mode, however there was no way to actually run them, at least not directly.
Browsing the file ‘/tmp/linux.log’ confirmed the device was actually based on Ambarella hardware, as indicated by any amount of Googling for “Paralenz Vaquita chipset”, which claims it is based on the H2s65 from Ambarella.
This meant we actually now had more than one way to get a shell on the camera. Within the file ‘/etc/init.d/S50service’ was a section near the bottom:
if [ -f /tmp/SD0/telnet/enable ]; then
#Make Linux console available on Telnet
telnetd
fi
SD0 seemed to imply the SD card, so getting telnet started would simply require the creation of a folder on the SD card called ’telnet’ with an empty file called ’enable’ within.
The downside of using telnet is that it requires an active wifi network on the camera. The wifi network deactivates as soon as the selector ring is switched out of the Settings menu, which limits what you can do. What was needed instead was a way of starting some of the other scripts in ‘/usr/local/share/script/’, such as ‘usb3_console.sh’, which seemed to set the USB port on the camera to appear as a USB serial device instead of a mass storage device. While you could simply activate the telnet service, switch on wifi and then run the script from there, this was highly inconvenient.
The other way to get a shell involves usage of the Ambarella RTOS auto execution script that runs on bootup. Ambarella chipsets are very commonly used for dash cameras and a multitude of action cameras, including the well known GoPro. As a result there is already a well established hacking scene for Ambarella based devices, which makes frequent mention of placing a file called ‘autoexec.ash’ on the root of the device SD card which gets executed on boot.
Much research later, a suitable set of commands to get a shell were found. Instead of using a flagged setting to start the telnetd on the camera, the autoexec approach allowed for execution of any command. I decided to go with USB serial, with the following autoexec.ash script:
sleep 10000
t ipc rpc clnt exec2 '/usr/local/share/script/usb3_console.sh'
Adding sleep 10000 to the start ensured the Linux subsystem (which runs in parallel but contained within the RTOS environment) was fully booted.
I now had a shell to access!
Even easier, simply using ‘root’ as the username dropped the session straight to a shell, giving full unrestricted access to the entire linux system of the camera.
As an aside, one of the first things I noticed when logging in on the shell was the hostname of the camera. Its rather cute actually, as the hostname of camera is “Eudyptula”, which is a genus of penguins of which there are two species (the ’little penguin’ and ‘Australian little penguin’). The name originates in the Greek, and translates to “Good little diver”. How fitting!
Brief investigation of the camera filesystem
The filesystem of the camera is quite similar to other Ambarella chipset devices, and there are actually a few stray references here and there to other Ambarella devices. Oddly, there is also a series of scripts which are clearly intended for operating LTE modems, so maybe there once was the intent to implement in-built LTE functionality down the track. This would probably have tied in well with the camera rental program, as the combination of LTE and GPS would be quite effective in discouraging theft, and the LTE data link would have allowed for dive data and video to be pushed to the Paralenz cloud quite seamlessly.
Reading ‘/proc/mtd’ identified volume labels for storage devices, most of which were only relevant to the underlying Ambarella systems, but which would be useful for further investigation later. Of particular interest are the mount points ‘/tmp/FL0’ and ‘/tmp/SD0’, which seem to be internal persistent memory and the SD card, respectively.
There is not much in ‘/tmp/FL0’, which appears to contain three files which indicate the version information of the camera, a directory containing cached GNSS data, another directory called ‘pref’ which contains wifi configuration, and of most interest - a file called settings.json which contains all the currently configured for the camera as defined through the user interface.
Compiled into the firmware read-only filesystem is the file pd2-version, located in /etc. This contains a single string that matches the firmware build version (shown on the original .plf file), which in my case is 22.32.21349. This appears to be used by pd2ui to display the firmware version on the OLED screen during boot.
Brief investigation of the camera firmware file
The firmware file (vaquita_22.32.21349.plf in my case) is a tar gzip file which contains two files, PDC2AAFW.bin and PDC2AAFW.md5. the MD5 file contains a hash of the binary file, and references the original build path of the same, in this case ‘releases/22.32.21349’. Updating the firmware of the camera involves decompressing the tar gzip file on the SD card, then using a verification process which matches the md5 hash of the supplied binary file with that in the md5 file to ensure its integrity prior to writing it to the onboard flash.
This process seems very similar to that used in other Ambarella based devices, so there is a high likelihood that these can be unpacked and repacked in much the same way. Further investigation will be needed to confirm this.
What is in /sys/kernel/camera?
This appears to be a specific part of the sysfs pseudo file system that provides a means of control between the linux environment and the actual camera hardware in the Vaquita. Below is a listing of its contents:
/sys/kernel/camera
├── cmd_capture
├── param_auto_exposure
├── param_video_codec
├── cmd_dsp_suspend
├── param_auto_whitebalance
├── param_video_format
├── cmd_record
├── param_eis
├── param_video_quality
├── dsp_state
├── param_exposure
├── param_whitebalance
├── is_capturing
├── param_still_format
├── param_zoomlevel
├── is_recording
└── param_still_raw
The vast majority of these are write-only, and do not provide any actual output, with the exception of ‘dsp_state’, ‘is_recording’ and ‘is_capturing’. These provide either a 0 or a 1 depending on state.
Conveniently, the camera also has strace included, which I suspect is because the firmware was still under heavy development up until the demise of Paralenz. The good thing about this is that it can be run to determine what is being written to each of the files in /sys/kernel/camera while the camera is under certain types of operation.
strace -fe trace=file,read,write -p <PID of pd2ui> -s 256 -P /sys/kernel/camera/<whichever file you want to monitor reads/writes for>
As usual with strace, you can simply stack -P options to monitor multiple files, or trace different types of activity depending on what you want to find out. Concentrating on the sysfs stuff provided the following findings:
/sys/kernel/camera/cmd_capture - Setting this to 1 manually triggers a capture event (a photo is taken)
/sys/kernel/camera/param_auto_exposure - Set to 1 when automatic exposure is used. Set to 0 when manual exposure is used. Manual exposure mode requires param_exposure to be set too.
/sys/kernel/camera/param_video_codec - Selects between AVC and HVC encoders. Does not get used by pd2ui at all but manually setting this to 0 or 1 selects AVC while 2 or above seems to select HVC encoder. As pd2ui doesn't ever seem to hit this setting, it can likely be set using a bootup script.
/sys/kernel/camera/cmd_dsp_suspend - This seems to set the DSP into a suspended state, however no normal mode uses this. Suspending the DSP disconnects the SD card and makes the screen, selector ring and trigger input completely unresponsive. Attempting to unsuspend the DSP by sending a 0 instead of a 1 to this file does nothing and the unit must be force rebooted by holding the trigger for at least 13 seconds.
/sys/kernel/camera/param_auto_whitebalance - Set to 1 when auto whitebalance is used. Set to 0 when a manual whitebalance is used. Manual whitebalance required param_whitebalance to be set too.
/sys/kernel/camera/param_video_format - Numerical values depending on selected video resolution and framerate (0 - 2160p60, 1 - 2160p50, 2 - 2160p30, 3 - 2160p25, 4 - 1520p60, 5 - 1520p50, 6 - 1520p30, 7 - 1520p25, 8 - 1080p240, 9 - 1080p200, 10 - 1080p120, 11 - 1080p100, 12 - 1080p60, 13 - 1080p50, 14 - 1080p30, 15 - 1080p25, 16 - 720p240, 17 - 720p200, 18 - 720p120, 19 - 720p100, 20 - 720p60, 21 - 720p50, 22 - 720p30, 23 - 720p25)
/sys/kernel/camera/cmd_record - Setting this to 1 starts a video recording. Setting it back to 0 stops the recording and saves the output. Recordings started in this way do not get reflected on the UI.
/sys/kernel/camera/param_eis - Explicitly set to 0, does not seem to ever get enabled through the UI. Can be manually set to 1, however this only seems to affect low framerate videos (720p and 1080p up to 60FPS) and does not impact 1520p or 2160p above 30FPS. Unknown whether EIS actually works or not, further testing is required. At minimum when enabled, this setting causes the outer edge of the image to be trimmed, presumably for use in stabilising the image.
/sys/kernel/camera/param_video_quality - Set to 1 when using Video or Snap mode, always set after param_video_format. Not sure what other options might be possible for this parameter.
/sys/kernel/camera/dsp_state - Output file, reads as 1 when DSP is active
/sys/kernel/camera/param_exposure - Set with three comma separated values based on configuration settings (settings.json). The first value is 'shutterSpeed', second is 'iso' and the third always seems to be 4096.
/sys/kernel/camera/param_whitebalance - Set with three comma separated values based on configuration settings (settings.json), but not with raw values as is the case with exposure. Only gets used when DCC is set to inactive. Values are as follows: 3500K - (4800,4096,12200), 5000K - (7500,4096,8500), 5600K - (8120,4096,7680), 6500K - (8800,4096,6600)
/sys/kernel/camera/is_capturing - Output file, reads as 1 when capturing, 0 when not capturing.
/sys/kernel/camera/param_still_format - Does not get used by pd2ui when selecting any particular mode or when configuring picture log or video settings. Experimenting by manually setting values results in output pictures of different resolutions when set between 0 and 4. Anything higher results in no output being generated. Resolutions as follows: 0 - 12MP 4:3 (4000x3000), 1 - 9MP 16:9 (4000x2250), 2 - 8.2944MP 16:9 (3840x2160 aka 4K), 3 - 0.9216MP 16:9 (1280x720 aka 720p).
/sys/kernel/camera/param_zoomlevel - Always seems to be set to 4 by pd2ui. Changing it to any other value does not seem to have any real impact on the output image.
/sys/kernel/camera/is_recording - Output file, reads as 1 when video is recording, 0 when video is not being recorded.
/sys/kernel/camera/param_still_raw - Explicitly set to 0 on Snap mode, also when settings are saved for custom modes on Picture Log, but NOT when the same custom mode is selected using the selector ring. Not sure why this is the case, but it does not impact behaviour nor recording of RAW format files when the SD card raw/enable file is present. Suspect this is not fully or correctly implemented. Manually setting to 1 also generates the same RAW format file as the SD card enable file does.
What is pd2ui?
Now that I had a shell, I could poke around in the camera and see what various things did. The first stop was an application in ‘/usr/bin/’ called ‘pd2ui’. This is started on boot under the init.d script ‘S98pd2ui’, which runs ‘/usr/bin/uilauncher’, which is simply a shell script wrapper around pd2ui. This application appears to be the core application which controls camera function.
The log file created under ‘/tmp/linux.log’ could actually instead be created on the SD card by creating a folder on the SD card root called ‘syslog’ containing an empty ’enable’ file, similar to enabling the telnet service. Unfortunately no such functionality existed for pd2ui UNLESS the application was crashed (which triggers the camera to reboot after a few seconds), in which case the log file would be gzipped and copied to the SD card, presumably for analysis by Paralenz support to determine the cause of the crash.
Interesting debugging/function flags
With shell access this was no major barrier, so instead the next step was to run the strings command against pd2ui and see what interesting human readable contents were present. This gave an enormous amount of output to peruse, which showed up additional debug SD card files such as ‘/tmp/SD0/raw/enable’ and ‘/tmp/SD0/charging/enable’, along with references to functions from shared libraries. Digging for strings in the shared libraries in turn identified additional similar files such as ‘/tmp/SD0/dcc/enable’, ‘/tmp/SD0/battery/enable’, ‘/tmp/SD0/temperature/enable’ and ‘/tmp/SD0/uclog/enable’.
Along with these enable flags were a few other files, for which the purpose was less clear…
'/tmp/SD0/dcc/parabolas'
'/tmp/SD0/battery/cutoff'
'/tmp/SD0/temperature/temperature'
Experimenting with the above files identified the following functionality:
'/tmp/SD0/raw/enable' - Causes snapshots to be saved in some sort of RAW format. Not sure exactly which.
'/tmp/SD0/dcc/enable' - Seems to record statistics for DCC in CSV files within '/tmp/SD0/dcc/'
'/tmp/SD0/battery/enable' - Logs battery statistics to CSV files
'/tmp/SD0/temperature/enable' - Logs either camera or water temperature statistics to CSV files
'/tmp/SD0/charging/enable' - Logs information on charging via USB port, including USB port voltage, battery voltage and thermal state.
'/tmp/SD0/uclog/enable' - Appears to log output from ucLog to file. Not quite sure where ucLog is being used though, further investigation required
Enabling these features writes additional data to the SD card, so it is possible that having them enabled may well compromise the ability for the camera to record video, or otherwise affect stability. Further testing will probably be needed before such behaviour can be confirmed.
The DCC logs appear to store data relating to the application of filtering to the recorded video or images, as described in the patent for the DCC functionality available here
How does pd2ui figure out what the device is doing?
As pd2ui lives inside the linux environment on the device, it needs a way to receive input from the RTOS environment letting it know what is going on. This all seems to be handled by ’libparalenz-uc.so’, which does the vast majority of this by simply listening to ‘/dev/ttyS0’. Reading directly from ttyS0 shows a consistent feed of data, being updated at least once a second (give or take). So far, it would appear each message has a header of ‘50 44’, with the next bytes defining both the source of the message and the content of the message itself. These messages can also be read using strace against pd2ui using the command ‘strace -xx -fe trace=file,read,write -p 259 -s 256 -P /dev/ttyS0’ (Don’t forget to add 2>&1 if you are piping the output to grep) which generally gives a much cleaner output than attempting to directly read ttyS0.
The changes in selector ring were identified as follows, with the header of ‘50 44 0A 01’:
Power - '50 44 0A 01 01 0C 21'
Snap - '50 44 0A 01 08 13 28'
Video - '50 44 0A 01 07 12 27'
Custom 1 - '50 44 0A 01 05 10 25'
Custom 2 - '50 44 0A 01 06 11 26'
Settings - '50 44 0A 01 04 0F 24'
DiveLog - '50 44 0A 01 03 0E 23'
GPS Position - '50 44 0A 01 02 0D 22'
Pulling and releasing the slider (trigger) on the camera generates similar messages, however there is more to decode due to the way the slider functions. The slider has two real states, pulled back, and not pulled back. The messaging adds more to that however, also seemingly indicating how long the slider has been pulled back for. Some messages were identified, all with the header of ‘50 44 03 05’:
Slider pulled back - '50 44 03 05 02 00 00 00 00 0A 3D'
Slider held for one second - '50 44 03 05 03 00 00 01 F4 00 38'
Slider held for two seconds - '50 44 03 05 03 00 00 03 E8 F6 30'
Slider held for three seconds - '50 44 03 05 03 00 00 05 DC EC 28'
Slider held for four seconds - '50 44 03 05 03 00 00 07 D0 E2 20'
Shorting the two exposed points on the top of the camera appears to affect the content of the message with the header of ‘50 44 0C 02’, which appears to be water salinity, or conductivity. Using metal to short the pins results in a message of ‘50 44 0C 02 00 58 66 8E’, while a damp or moist finger causes the final three bytes to vary depending on pressure and apparent conductivity. Corresponding values were found in the LOGS CSV files, where the metal short indicated a “salinity” of 88, which matches the hex value of ‘00 58’. The damp/moist finger gave a lower reading of between 1 and 3, also reflected in the message in a similar fashion. Further testing with water of varying salinity will likely help map the message structure.
The message with header ‘50 44 07 02’ is related to device temperature. As the temperature gradually rises, the messages change. Below is an example of the temperature rising slightly on the device: Original - ‘50 44 07 02 01 3E 48 62’ Increased - ‘50 44 07 02 01 3F 49 63’
The message with header ‘50 44 06 04’ is related to pressure. This is harder to test as there are no particularly easy ways to vary pressure in a controlled manner while the Vaquita is plugged up to a USB terminal, however triggering a depth calibration caused the message to change to ‘50 44 06 04 00 00 00 00 0A 38’ from a previous value (at the same temperature and pressure) of ‘50 44 06 04 00 00 00 0A 14 42’. It is possible for the pressure value to underflow as well due to being calibrated at a higher pressure prior to the pressure dropping, for example running the calibration on a day where the barometric pressure is high, or at a lower altitude before travelling to a higher one. This results in a message of ‘50 44 06 04 FF FF FF FF 06 2E’ or similar, and appears to cause problems with pd2ui when rendering the dive log. It would seem a good course of action would be to calibrate the depth sensor immediately prior to diving to ensure more consistent behaviour.
The pressure sensor appears to be located underneath or behind the slider.
Further investigation of pd2ui.log
The log file for pd2ui gives a remarkable amount of information. When pd2ui starts, it launches a number of additional threads to perform various tasks, such as managing DCC, connecting to and watching the filesystem on the SD card, collecting information from the camera hardware such as battery level and temperature, enabling bluetooth remote control, synchronising RTC time with the time source from the GPS device, and drawing relevant information to the OLED display screen.
A number of startup processes run, which includes scanning for firmware updates on the SD card and initiating a firmware upgrade if required.
Also logged are various interactions with user input. Each input which elicits a vibration feedback buzz is recorded as ‘[ParalenzPlatform] vibrate’, while rotating the selector ring records each selected state in integer form. These correspond as follows:
Power - 1
Position - 2
Dive Log - 3
Settings - 4
Custom1 - 5
Custom2 - 6
Video - 7
Snap - 8
Interestingly, the order of these modes does not match that on the actual selector ring. Instead it follows the sequence (when going clockwise) Power, Position, Dive Log, Settings, Custom2, Custom1, Video, Snap (or 1, 2, 3, 4, 6, 5, 7, 8). This would suggest at some point the original design decision had these two options inverted, and it was necessary to flip the positions in code to match what ended up being marked on the physical selector ring.
pd2ui also connects to the RTOS environment to capture when events take place, such as when video is recorded and when stills are captured. Actual video and still image recording appears to take place outside of the Linux environment. An RTSP service is run locally, however no apparent purpose for this has been found so far, although it might potentially be related to liveview or streaming the videos already recorded on the device. This functionality used to exist for the Dive Camera+ in the old Paralenz Dive app, however the new Android app only seems to support downloading the videos and doesn’t appear to have liveview at all for either the Dive Camera+ or Vaquita.
What is in settings.json
The file /tmp/FL0/settings.json contains all of the settings as defined by pd2ui. Through trial (setting each setting via the UI and checking the difference in settings.json), the following options were identified:
"autoOff": (Can be 0 for disabled, 1 for 10 minutes, 2 for 20 minutes)
"autoRecordMode": (0 - Off, 1 - 0.5M, 2 - 1.5M, 3 - 3M, 4 - 6M),
"createBlurHashes": (Always set to true, generates blurhashes under /tmp/SD0/DCIM/100PRLNZ/.meta)
"custom1Burst": (Always seems to be set to 1, seems to be unimplemented)
"custom1PictureLog": (0 - 1P/5S, 1 - 1P/10S, 2 - 1P/60S, 3 - 1P/120S, 4 - 1P/S, 5 - 1P/2S, 6 - 2P/S )
"custom1SlowMotionFPS": (Always set to 3, seems to be unimplemented)
"custom1SlowMotionRes": (Always set to 1, seems to be unimplemented)
"custom1TimelapseRes": (Always set to 1, seems to be unimplemented)
"custom1TimelapseSpeed": (Always set to 0, seems to be unimplemented)
"custom1VideoFPS": (0 - 30FPS, 1 - 60FPS, 2 - 120FPS, 3 - 240FPS, 4 - 25FPS, 5 - 50FPS, 6 - 100FPS, 7 - 200FPS)
"custom1VideoRes": (0 - 720p, 1 - 1080p, 2 - 1520p, 3 - 2160p)
"custom2Burst": (Always seems to be set to 1, seems to be unimplemented)
"custom2PictureLog": (0 - 1P/5S, 1 - 1P/10S, 2 - 1P/60S, 3 - 1P/120S, 4 - 1P/S, 5 - 1P/2S, 6 - 2P/S )
"custom2SlowMotionFPS": (Always set to 3, seems to be unimplemented)
"custom2SlowMotionRes": (Always set to 1, seems to be unimplemented)
"custom2TimelapseRes": (Always set to 1, seems to be unimplemented)
"custom2TimelapseSpeed": (Always set to 0, seems to be unimplemented)
"custom2VideoFPS": (0 - 30FPS, 1 - 60FPS, 2 - 120FPS, 3 - 240FPS, 4 - 25FPS, 5 - 50FPS, 6 - 100FPS, 7 - 200FPS)
"custom2VideoRes": (0 - 720p, 1 - 1080p, 2 - 1520p, 3 - 2160p)
"customMode1": (0 - Video mode, 4 - Picture Log mode)
"customMode2": (0 - Video mode, 4 - Picture Log mode)
"dateFormat": (Always set to 0, does not seem to be present on Date/Time menu in settings)
"dccActive": (Either 0 - Inactive or 1 - Active)
"dccMode": (Always set to 5, seems to be the automatic DCC mode as there is no option for blue or green)
"depthUnit": (0 - Meter, 2 - Feet)
"exposureMode": (0 - Auto Exposure, 2 - Manual Exposure)
"iso": (Value is 10 times the ISO number, ie ISO-100 is 1000, ISO-200 is 2000, ISO-3200 is 32000, etc)
"language": (0 - English, 2 - Russian, 3 - Chinese, 4 - German, 8 - Korean)
"maxSnapDuration": (Always set to 120, no option to change in menu)
"overlayActive": (0 for inactive, 1 for active - Not always active in certain video modes such as 4K)
"overlayLine1": (0 - Depth, 1 - Temperature, 2 - Ascent/Descent Speed, 3 - Date, 4 - Time of Day, 5 - Date and Time, 6 - Empty Line, 7 - Dive Time)
"overlayLine2": (0 - Depth, 1 - Temperature, 2 - Ascent/Descent Speed, 3 - Date, 4 - Time of Day, 5 - Date and Time, 6 - Empty Line, 7 - Dive Time)
"shutterSpeed": (33300 - 1/30, 16600 - 1/60, 8300 - 1/120, 4160 - 1/240, 2000 - 1/500, 1000 - 1/1000, 500 - 1/2000, 250 - 1/4000, 125 - 1/8000
"stillFormat": (Always set to 0, no option to change in menu)
"stillRes": (Always set to 2, no option to change in menu)
"temperatureUnit": (Either 0 for Celcius or 2 for Fahrenheit. No idea what 1 might be?)
"timeFormat": (0 - AM/PM, 1 - 24H)
"timeZone": (0 - UTC-12, 1 - UTC-11, 2 - UTC-10 , 3 - UTC-9:30, 4 - UTC-9, 5 - UTC-8, 6 - UTC-7, 7 - UTC-6, 8 - UTC-5, 9 - UTC-4, 10 - UTC-3:30, 11 - UTC-3, 12 - UTC-2, 13 - UTC-1, 14 - UTC-0, 15 - UTC+1, 16 - UTC+2, 17 - UTC+3, 18 - UTC+3:30, 19 - UTC+4, 20 - UTC+4:30, 21 - UTC+5, 22 - UTC+5:30, 23 - UTC+5:45, 24 - UTC+6, 25 - UTC+6:30, 26 - UTC+7, 27 - UTC+8, 28 - UTC+8:45, 29 - UTC+9, 30 - UTC+9:30, 31 - UTC+10, 32 - UTC+10:30, 33 - UTC+11, 34 - UTC+12, 35 - UTC+12:45, 36 - UTC+13, 37 - UTC+14)
"videoEIS": (Always set to 0)
"videoEncoding": (Always set to 1)
"videoFPS": (0 - 30FPS, 1 - 60FPS, 2 - 120FPS, 3 - 240FPS, 4 - 25FPS, 5 - 50FPS, 6 - 100FPS, 7 - 200FPS)
"videoQuality": (Always set to 1)
"videoRes": (0 - 720p, 1 - 1080p, 2 - 1520p, 3 - 2160p)
"whitebalance": (1 - 3500K, 2 - 5000K, 3 - 5600K, 4 - 6500K)
"zoomLevel": (Always set to 4, seems to be unimplemented)
Investigating the RAW format output
This proved to be a bit of a challenge, and is still somewhat ongoing, as I have not previously worked with raw files before. That said, here is what has been found so far…
The Vaquita uses a Sony IMX577, which is a 12.3 mega-pixel sensor capable of outputting 4K video at up to 60FPS, and 1080p video at up to 240FPS. The sensor itself has an effective resolution of 4072x3064, however the active pixel area is actually 4056x3040. In a practical sense, the actual raw output is slightly smaller than this, measuring in at 4032x3024 (4:3 ratio multiplied by 1008), with pixel data stored in 12-bit elements, potentially padded out and stored to file as 16 bits. Apparently the Bayer Color Filter Array for the sensor is BGGR, however this is based on various online discussion threads largely for the Raspberry Pi HQ camera, which uses the similar Sony IMX477 sensor. Further investigation is required before these images can be properly decoded, and to determine what other parameters have an impact on the image that gets recorded.
There may potentially be functionality to record raw video as well, however such functions seem to be incomplete or may be code fragments that are not implemented. No means has yet been identified to attempt to record any sort of raw video format, and it is expected if it is possible, it would only likely work for lower resolutions and framerates due to the huge bandwidth requirements.
The raw images can be decoded using the python script below (credit to Nikolaj Bech Andersen!)
import numpy as np
import cv2
import sys
path = sys.argv[1]
is_wide = True
imcols = 4032
imrows = 2268 if is_wide else 3024
imsize = imrows*imcols
with open(path, 'rb') as rawimage:
img = np.fromfile(rawimage, np.dtype('>u2'), imsize).reshape((imrows, imcols))
rgb = cv2.cvtColor(img, cv2.COLOR_BAYER_BG2BGR)
cv2.imwrite(path + '.TIFF', rgb)
This will take an input raw format still image and give an output tiff file of approximately 32mb. In my test image I found the resulting image was quite dark and some of the colours were slightly off, but this was very easily rectified in even the most basic image editing software. More experienced graphic designers and people familiar with Photoshop likely could take this a lot further.
Investigating the EIS functionality
EIS functionality appears to exist in some form, however it is unclear as to how effective it is, or whether it works as described. Some testing with various video modes showed video distortion typical of functioning EIS, but only for certain resolutions and framerates, these being at 720p or 1080p at a framerate of either 25, 30, 50 or 60 FPS, and 1520p or 2160p at a framerate of 25 or 30 FPS. Other framerate and resolution combinations showed no such distortion. Some decidedly non-scientific testing (which consisted of putting the camera on a tripod and thumping it a bit) indicated at least a level of image stabilisation was taking place at 2160p30, however faster panning motions with the camera were not compensated for, and the low framerate lead to significant blurring of the video itself.
Within the binaries on the RTOS partition in memory, there is substantial referencing to EIS, including a number of t commands for ambashell. Unfortunately these do not feed output to stdout, so gathering information from them has been difficult so far.
The reason for EIS not functioning at the higher framerates of each video mode is likely due to the computational cost of running stabilisation over such a quantity of data, as the higher the resolution and framerate, the more processing power that is required to keep everything in sync.
Whether the EIS function works or not, in many cases it may make more sense to record at the highest quality and framerate possible that suits your purpose, then post-process the video on a suitable editing application to apply a level of stabilisation instead.
Accessing the OLED screen
The OLED screen of the Vaquita appears to be presented as a standard linux framebuffer device. An lsof identified that pd2ui had /dev/fb0 open, and subsequently running ‘cat /dev/urandom > /dev/fb0’ wrote the expected random pixel snowstorm to the OLED screen. Checking dmesg identified the OLED screen as being an RM67160, attached to an SPI interface.
Running fbset dumped the following output:
mode "180x120-0"
# D: 0.000 MHz, H: 0.000 kHz, V: 0.000 Hz
geometry 180 120 180 120 16
timings 0 0 0 0 0 0 0
accel false
rgba 5/11,6/5,5/0,0/0
endmode
So does this mean it could run Doom?
In theory, yes! However… getting it to actually display anything is another matter entirely! The problem lies with the framebuffer resolution and the fact that Doom’s native resolution is 320x240. As the native display of the RM67160 is 180x120, it won’t fit. The various versions of FBDoom that I found and played around with had no capacity to scale below the native resolution, which means quite a bit more hacking around would be required to make this work. What scaling they did have only operated with integer values, so the autoscaling functions would identify the native resolution of the framebuffer device as being below 1, and would default to zero, hence displaying absolutely nothing.
Another option might be to feed the output to a video processor to scale it down that way before feeding it to the screen instead, similar to how the current video streams from the camera are handled, however this would require a fair bit more investigation.
More about the MTD on the Vaquita
MTD in linux means Memory Technology Device, and usually refers to flash memory devices. These are at the core of pretty much any linux based embedded device such as the Paralenz Vaquita and many other Ambarella SoC based cameras, as this is where everything that controls the device and makes it bootable is stored. The contents of the flash are presented in various ways, and when investigating the contents of ‘/proc/mtd’ earlier while exploring the filesystem, I found the following:
dev: size erasesize name
mtd0: 00020000 00020000 "VENDOR_DATA"
mtd1: 02000000 00020000 "SYS_SW"
mtd2: 00700000 00020000 "DSP_uCode"
mtd3: 05000000 00020000 "SYS_DATA"
mtd4: 00a00000 00020000 "LINUX_Kernel"
mtd5: 03c00000 00020000 "LINUX_RFS"
mtd6: 01e00000 00020000 "LINUX_HIBER_IMG"
mtd7: 00d80000 00020000 "VIDEO_REC_IDX"
mtd8: 00280000 00020000 "CALIB"
mtd9: 001c0000 00020000 "USER_SETTING"
mtd10: 00680000 00020000 "DRIVE_A"
mtd11: 00680000 00020000 "DRIVE_B"
This gives us some clues as to what is stored where, and some targets for further investigation. The two final devices (mtd10 and mtd11) are the on-board writable memory, typically used to store small files that get updated as part of operational usage and which are used between the RTOS and linux environments. ‘DRIVE_A’ is also visible via the RTOS environment as ‘a:’ and is mounted in the linux environment through this as a filesystem of type ‘ambafs’ on ‘/mnt/FL0’. This is where camera settings, GPS location data and wifi preferences are stored as previously mentioned, and can also be where an autoexec.ash file is placed to be executed by the RTOS on camera bootup. One trick I’ve seen used before is to put an autoexec.ash on this partition to copy or write a small file to whatever SD card is placed in the camera with information on who the device belongs to. This way if it is lost, stolen or the SD card removed and replaced, the information will be recreated on every single bootup.
In the case of ‘DRIVE_B’, this appears to be completely unused in the Vaquita, and contains only some data for a FAT12 filesystem without any actual contents.
A matter of time and position - The GPS module
The Vaquita comes equipped with a ublox m8 generation GPS/GNSS module (More details over on the ublox product page). Support for the device is built into the linux kernel using the module ublox_m8_i2c.ko, indicating it is connected using i2c and presented as /dev/ublox_m8. Reading the device using cat, or by running strace against pd2ui shows the typical output consists of standard NMEA messages, mostly $GNGGA lines. To learn more about these refer to the NMEA page on the gpsd documentation site. Initially after booting up, the module will return lines at the rate of one per second (1Hz) such as below:
$GNGGA,012828.00,,,,,0,00,99.99,,,,,,*79\r\n
These are quite typical while the GPS has not established a fix, indicated by the ‘0,00’ section, which as described on the gpsd documentation indicates zero satellites are in use and a signal quality of zero, meaning a fix is not available. If monitored for long enough, eventually satellites will start being acquired, with sufficient numbers providing a fix and latitude and longitude values.
Unfortunately it often takes quite a while for the GPS system to establish a fix. While this can be due to poor sky visibility, another potential cause is the module is doing a cold start, requiring it to reacquire the full almanac and ephemeris, which can take 15 minutes or more.
When it comes to date and time settings on the Vaquita, this relies on a combination of the on-board Ambarella RTC and the GNSS module RTC, and there is no way for it to be set through the settings available via the standard user interface. Instead, when pd2ui starts up a sync process gets run which sets the correct system time and adjusts internal clocks to match the date and time specified by the GNSS module. This can be seen early in ‘/tmp/pd2ui.log’, where a number of messages similar to the following are recorded:
[TimeService] TimeService::TimeService(M8*, ParalenzUC*, QObject*)
[TimeService] systemTimeDriftFromUc( 10 )
[TimeService] Offset too small
[TimeService] systemTimeDriftFromGNSS( -10131 )
[TimeService] param_unix_time: ' "1674262916" '
[TimeService] Updating system time to 1674262916
It would appear that it is possible to set a time manually with some fiddling by feeding a unix time to ‘/sys/kernel/rtc/param_unix_time’, however this would invariably be overwritten by whatever the GNSS module RTC had recorded on the next bootup. The best option for keeping the correct time on the Vaquita would appear to be powering the camera up with auto power-off disabled in a location it can get a clear view of the sky and waiting the required length of time for a GPS fix to be established. This will provide an accurate time reference which will be used to sync the GNSS module RTC, which then will cascade down to the Ambarella RTC and linux system time. To be absolutely sure, power the camera off and back on once a GPS fix has been made.
Wi-Fi on the Vaquita
Wi-Fi is activated on the Vaquita through the Settings menu by selecting Wi-Fi. Enabling it displays an overlay on the screen which says “Connect on device”, and runs a series of commands via pd2ui. From the pd2ui.log file we can see the following when Wi-Fi is enabled, then disabled: (note MAC address has been redacted)
[WifiControl] BEGIN_WIFI_START
[WifiControl] MOUNT_FTP_FOLDER
[ParalenzPlatform] vibrate
[FWUpdateService] firmwareFileChecker
[FWUpdateService] Checking for files
[WifiControl] CHANGE_OWNERSHIP_OF_FTP_FOLDER
[FWUpdateService] firmwareFileChecker
[FWUpdateService] Checking for files
[WifiControl] PRELOAD_DRIVER
[WifiControl] LOAD_DRIVER
[WifiControl] GET_MAC_ADRESS
[WifiControl] CONFIGURE_WPA_SUPPLICANT
[WifiControl] Got string: "lo Link encap:Local Loopback \n inet addr:127.0.0.1 Mask:255.0.0.0\n UP LOOPBACK RUNNING MTU:65536 Metric:1\n RX packets:6 errors:0 dropped:0 overruns:0 frame:0\n TX packets:6 errors:0 dropped:0 overruns:0 carrier:0\n collisions:0 txqueuelen:1 \n RX bytes:328 (328.0 B) TX bytes:328 (328.0 B)\n\nwlan0 Link encap:Ethernet HWaddr 08:E9:F6:AE:83:2E \n BROADCAST MULTICAST MTU:1500 Metric:1\n RX packets:0 errors:0 dropped:0 overruns:0 frame:0\n TX packets:0 errors:0 dropped:0 overruns:0 carrier:0\n collisions:0 txqueuelen:1000 \n RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)\n\n"
[WifiControl] Pos: 385
[WifiControl] Mac adress is: "RE:DA:CT:ED:00:00"
[WifiControl] Cut mac is "REDACTED0000"
[WifiControl] CONFIGURE_WLAN0
[WifiControl] START_WPA_SUPPLICANT
[WifiControl] SETUP_DNS
[WifiControl] WIFI_START_DONE
[ParalenzPlatform] vibrate
[ParalenzPM] ucChargerInterface
[WifiControl] BEGIN_WIFI_STOP
[WifiControl] SHUTDOWN_WIFI
[ParalenzPlatform] vibrate
[WifiControl] TERMINATE_WLAN0
[WifiControl] SHUTDOWN_WLAN0
[WifiControl] KILL_PROCESSES
[WifiControl] REMOVE_FILES
[WifiControl] REMOVE_DRIVER
[WifiControl] WIFI_STOP_DONE
This gives a fairly good run-down on what the camera is doing when enabling and disabling Wi-Fi, specifically setting up the directory and permissions for where files are meant to be read and written from and to, loading the Wi-Fi drivers, gathering relevant information from the system (namely MAC address, which gets used to define the AP SSID), then configuring the interface wlan0 and starting wpa_supplicant and dnsmasq which serves as the DHCP server for the network.
Under the hood, a bit more is happening. When ‘BEGIN_WIFI_START’ occurs, the camera also writes a file to ‘/tmp/SD0/.camera.json’. This file contains some basic configuration settings, the firmware version, the time and timezone, an ID and a count of photos and videos. It would appear this file is meant to be used by the Paralenz app or similar clients on connection, and changes or updates from the client application are written to ‘/tmp/SD0/.modify.settings’.
Similarly, a file also gets written to ‘/tmp/FL0/camera_data’ containing similar values, between ‘CONFIGURE_WPA_SUPPLICANT’ and ‘CONFIGURE_WLAN0’, as does the file ‘/tmp/wpa_supplicant.ap.paralenz.conf’, which holds the configuration for the Paralenz wireless network.
All of this functionality seems to be implemented through the shared library libparalenz-wl.so located in /usr/lib, which also contains the bluetooth low energy functionality.
It would appear this was under development at the time version 22.32.21349 of the firmware was released. While the library appears to have support for Wi-Fi STA mode, there does not appear to be any means by which to enable it in pd2ui, as the settings needed such as network and PSK are not recorded in any of the persistent settings files on either the SD card or on the onboard flash memory. In addition, the BLE functionality appears to be only partially complete. The camera dumps enormous amounts of debug data when a client pairs with the presented bluetooth device, and the last release of the Paralenz mobile application does not have any bluetooth functionality at all.
To gain STA functionality will likely require the use of autoexec.ash scripts to start Wi-Fi at boot, or development of a custom program which could watch for certain messages from the RTOS as transmitted on ‘/dev/ttyS0’, such as rotating from Custom2 to Settings (Selector ring state 6 to 4) three times in quick succession.
What comes next?
There is still a lot to do. For one thing, it would be nice to understand the workings of pd2ui better!
In addition to that, some other next steps are to identify and figure out how the overlay system works, specifically where the depth information is being derived from on the device and whether there are any other sensors which could be accessed as well.
It would also be nice to get into the RTOS terminal and investigate its commands further. It can be accessed to an extent using autoexec.ash, however some of the commands do not pipe output to stdout or stderr, making them difficult to log. The easiest approach would be to disassemble a Vaquita and hook an analyser to the traces on the mainboard, however that would almost certainly compromise the seals and make it impossible to use without risk of flooding, so I have no intention of doing that to my camera - not only would it be expensive, it would be likely impossible to even source a replacement at this point.
Some of the features discussed earlier are still a bit of a mystery, or rather poorly understood. These will need further investigation and testing to try and figure out what they do (such as parabolas for DCC), or to find their limitations (as is the case with EIS, which seems to sort of work, but to which the extent and effectiveness is unknown).
Further investigation of the flash onboard the camera, and of the filesystem will be ongoing. There is still a lot to learn yet!