This website has recently been off-line for just over two days due to, in this case, poor administration. Hands up! my error/s.

I am running this site on a server running SuSE Leap 15.0 with the latest ‘btrfs’ file system, and was looking to upgrade to the latest SuSE fixes release, 15.1.

Having run non-absolute cpio backups on /etc, /root, /var/lib, /srv and a couple of third party apps and settings in /usr/local, I then transferred all images to my backup system as usual, using FileZilla FTP client set in ‘Auto’ mode as usual.

Happy that there was enough disk space on my root partition to perform the upgrade I commenced manually downloading all updates to the release of 15.0 using ‘Zypper’.  The amount of updates required amounted to a couple of hundred, but all went smoothly, so proceeded to go through the procedure to manually upgrade the OS from 15.0 to 15.1 using ‘Zypper’ in ‘init 3’ mode.

All went well downloading and installing, until……………No space left on the ‘root’ partition halted the procedure.

1: I had totally forgotten to take into account that two major ‘btrfs snapshots’ would be taken during this procedure, dramatically reducing my free diskspace……….Lesson learnt.

I attempted to use a snapshot to boot the machine to it’s former state, you guessed correctly, it failed. However, I booted the machine to a command prompt and cleared all temporary file locations and proceeded to transfer and re-install via Filezilla each and every backup in turn. Thinking that would do the trick, I rebooted, except it would not reboot, nor was I able to rescue the system.

There was no real successful alternative but to rebuild the system to SuSE Leap 15.0 again and apply all updates prior to loading my back-ups. This I did.

Following the rebuild, I fortunately in this case reloaded the ‘/srv’ backup first. The ‘cpio’ archive read back with ‘cpio: warning: skipped 49 bytes of junk’ errors on virtually every backed up file. I checked all the archives and found that they were in the format of  ‘ASCII cpio archive (SVR4 with no CRC)’.

2: Never ever forget to test your transferred binary archive via ftp in binary mode and not ascii,once on the target backup system. What puzzles me is that my FileZilla transfer mode is set to ‘auto’, so it is obvious that auto is not working correctly……..Lesson learned.

After shouting obscenities from the rooftop of my house, not being a programmer and knowing that ‘shell script’ files would not help me here, I spent a good day testing applications trying to convert my Ascii to their former state.

SUCCESS.

The below python script written by Carl Mai saved my site. There is a small bug in the script at the moment where the program loops once completed. I believe this is being looked at. However, here is the script I used.

For info: One of my backups was 200meg containing thousands of files. This took nineteen hours to complete……So, give the script time to complete….It will, believe me.

Script:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from glob import glob
import itertools
import os
import sys

def run(folder):
for file in glob(folder + "/*"):
filename, file_extension = os.path.splitext(file)
with open(file, "rb") as fp:
data = fp.read()
# if this line prints "file, 1, 2, 1" everything should be fine..
# if this line prints "file, 1, 0, 0" it is probably corrupted by ftp
print(file, data.count(b"\n"), data.count(b"\r\n"), data.count(b"\r"))
if data.count(b"\r\n") > 0:
data = data.split(b"\r\n")
else:
data = data.split(b"\n")
c = 0
for join in itertools.product((b"\n", b"\r", b"\r\n"), repeat = len(data) - 1):
c+=1
print("%03d %s"%(c, str(b":".join(join))[2:-1]))
new_data = data[0]
for idx, j_str in enumerate(join):
new_data += j_str + data[idx+1]
with open("%s_%03d%s" % (filename, c, file_extension), "wb") as fp:
fp.write(new_data)

if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: ./fix.py foldername")
run(sys.argv[1])

README

bin ascii fixer

In cases where a binary file is transfered in ascii format via FTP. This fixer will brute force all possible original files.

What happens when transferring a binary file in ASCII mode

Since this kind of transfer will replace any of the following

\n -> \r\n (unix to windows)
\r -> \n (windows to unix #1)
\r\n -> \n (windows to unix #2)

we can not know 100% what the original file looked like.

Examples (where people had the same problem)

• https://superuser.com/questions/195612/recovering-corrupted-files-uploaded-in-wrong-ftp-mode
  • first answer is actually how I implemented this
• http://blog.deepcore.gr/?p=177
  • “fixgz” did not work for me
• https://www.inmotionhosting.com/support/website/file-management/corrupt-file-ftp-transfer
  • how to avoid (but still happened to me in filezilla with auto)

How to Use

./fix.py folder

Example Data

Inside demo_data are two images. Just run ./fix.py demo_data and then do ls demo_data.