Windows Newlines Will Kill Your Linux Scripts

Here’s a little something I give to my online Ruby and Python students every year.

Sometimes you create Ruby or Python code looks perfect, yet it will not run when you upload it to the server. You will get mysterious error messages like this:

Server error!

The server encountered an internal error and was unable to complete your request.

Error message:

Premature end of script headers: lab1.2.py

So, you look at your script, and all you can see is absolute perfection, but it still won’t work, so you run it on the command line, like this:

python3 lab1.2.py
and it runs perfectly. But, when you run it like this:
./lab1.2.py

… it goes down in flames:

unable to exec joesmith88/public_html/lab1.2.py: No such file or directory

What’s Going On?

What’s going on is that you really do have a fatal error in your code, and it’s an error that you can’t see. In fact, it’s invisible. The error is that you have uploaded a file that you created on a Windows machine.

There’s nothing intrinsically wrong with Windows, but Windows editors often, by default, do something that breaks — fatally breaks — CGI scripts that run on a Unix server: they add Windows line endings to the end of every line.

What Are Windows Line Endings?

Windows uses two characters for line endings: a carriage return and a new line. We would write it in Python like this: “\r\n“.

Unix Line Endings

Unix uses one character for line endings: a new line, like this: “\n“.

Why Do WIndows Line Endings Break CGI Scripts?

Assume that you upload a file that contains the code below. I’ll make the line endings conspicuous by using “\r\n” to indicate them:
#!/usr/local/bin/python3\r\n
print(’Content-type:text/html\n’)\r\n
print(’Hello, world’)\r\n
The real problem is the shebang line.
#!/usr/local/bin/python3\r\n

Here’s how Linux sees that line. First, Linux doesn’t know about Windows line endings; the “\r” character is just another character, like an “a”, “b”, or “c”. The shell sees the shebang and looks for a program named /usr/local/bin/python3\r.

Since Linux the “\r” as just another character in the program’s name, it looks on the files system for a file named /usr/local/bin/python3\r. Of course, that file doesn’t exist, so you get a mysterious fatal error, the error caused by the invisible “\r” character.

od (octal dump)

Your can see all of the characters in your scripts by using the od

program to dump all of the characters in your script.

Here’s an example that shows how to use it. This example dump comes from a script a student uploaded from a Windows machine. The “cr” and “nl” characters in the output represent the “\r\n” Windows line endings, and they appear at the end of every line of code. As we’ve already discussed, the “cr” will cause fatal errors when they are at the end of the shebang line. In a Unix CGI script, there should not be ANY cr characters unless you put them there intentionally.

% od -a joesmith88/public_html/lab1.2.py
0000000   #   !   /   u   s   r   /   l   o   c   a   l   /   b   i   n
0000020   /   p   y   t   h   o   n   3  cr  nl   #  sp   N   a   m   e
0000040   :  sp   J   o   s   e   p   h  sp  sp   S   m   i   t   h  cr
0000060  nl   #  sp   F   i   l   e  sp   n   a   m   e   :  sp   l   a
0000100   b   1   .   2   .   p   y  cr  nl   #  sp   D   a   t   e   :
0000120  sp   J   u   n   e  sp   2   1   ,  sp   2   0   1   1  cr  nl
0000140  cr  nl   #  sp   *   *   *   *   *   *   *   *   *   *  sp   L
0000160   a   b  sp   1  sp   E   x   e   r   c   i   s   e  sp   1   .
0000200   2  sp   *   *   *   *   *   *   *   *   *   *  cr  nl  cr  nl
0000220   p   r   i   n   t   (   ’   C   o   n   t   e   n   t   -   t
0000240   y   p   e   :  sp   t   e   x   t   /   h   t   m   l   \   n
0000260   ’   )  cr  nl   p   r   i   n   t   (   ’   \   n   ’   )  cr
0000300  nl   p   r   i   n   t   (  sp   ’   *   ’  sp   *  sp   1   0
0000320  sp   +  sp   ’  sp   L   a   b  sp   E   x   e   r   c   i   s
0000340   e  sp   1   .   2  sp   ’  sp   +  sp   ’   *   ’  sp   *  sp
0000360   1   0  sp   )  cr  nl   p   r   i   n   t   (   ’   H   e   l
0000400   l   o   ,  sp   w   o   r   l   d   !   ’   )  cr  nl  cr  nl

The Solution

The solution, fortunately, is really simple: tell your Windows or Mac editor (yes, even Mac editors misbehave), to stop using Windows-style line endings. Fix it right now. Dig into your editor’s configuration file and tell it to ALWAYS use Unix-style line endings. End of headache…

FTP Causes Problems, Too

Some FTP clients add a “\r\n” line ending by default. If you upload scripts that you know have Unix-style line endings, and they still fail, check them with od to see whether FTP has added Windows line endings. If so, configure your FTP client to behave properly.

Keep hacking…

[ Filed under Linux, Nostalgia, Python ]

Comments are closed.