`StartProject.test_custom_project_template_from_tarball_by_url` fails under Windows

[Aymeric Augustin]

======================================================================
FAIL: test_custom_project_template_from_tarball_by_url (regressiontests.admin_scripts.tests.StartProject)
Make sure the startproject management command is able to use a different project template from a tarball via a url
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Documents and Settings\Administrator\Desktop\django-trunk\tests\regressiontests\admin_scripts\tests.py", line
 1438, in test_custom_project_template_from_tarball_by_url
    self.assertNoOutput(err)
  File "C:\Documents and Settings\Administrator\Desktop\django-trunk\tests\regressiontests\admin_scripts\tests.py", line
 171, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains '['Traceback (most recent call last):', '  File "C:\\Documents
 and Settings\\Administrator\\Desktop\\django-trunk\\django\\bin\\django-admin.py", line 5, in <module>', '    managemen
t.execute_from_command_line()', '  File "C:\\Documents and Settings\\Administrator\\Desktop\\django-trunk\\django\\core\
\management\\__init__.py", line 420, in execute_from_command_line', '    utility.execute()', '  File "C:\\Documents and
Settings\\Administrator\\Desktop\\django-trunk\\django\\core\\management\\__init__.py", line 359, in execute', '    self
.fetch_command(subcommand).run_from_argv(self.argv)', '  File "C:\\Documents and Settings\\Administrator\\Desktop\\djang
o-trunk\\django\\core\\management\\base.py", line 196, in run_from_argv', '    self.execute(*args, **options.__dict__)',
 '  File "C:\\Documents and Settings\\Administrator\\Desktop\\django-trunk\\django\\core\\management\\base.py", line 232
, in execute', '    output = self.handle(*args, **options)', '  File "C:\\Documents and Settings\\Administrator\\Desktop
\\django-trunk\\django\\core\\management\\commands\\startproject.py", line 32, in handle', "    super(Command, self).han
dle('project', project_name, target, **options)", '  File "C:\\Documents and Settings\\Administrator\\Desktop\\django-tr
unk\\django\\core\\management\\templates.py", line 108, in handle', '    base_subdir)', '  File "C:\\Documents and Setti
ngs\\Administrator\\Desktop\\django-trunk\\django\\core\\management\\templates.py", line 189, in handle_template', '
return self.extract(absolute_path)', '  File "C:\\Documents and Settings\\Administrator\\Desktop\\django-trunk\\django\\
core\\management\\templates.py", line 261, in extract', '    archive.extract(filename, tempdir)', '  File "C:\\Documents
 and Settings\\Administrator\\Desktop\\django-trunk\\django\\utils\\archive.py", line 49, in extract', '    Archive(path
).extract(to_path)', '  File "C:\\Documents and Settings\\Administrator\\Desktop\\django-trunk\\django\\utils\\archive.p
y", line 57, in __init__', '    self._archive = self._archive_cls(file)(file)', '  File "C:\\Documents and Settings\\Adm
inistrator\\Desktop\\django-trunk\\django\\utils\\archive.py", line 128, in __init__', '    self._archive = tarfile.open
(file)', '  File "C:\\Python27\\lib\\tarfile.py", line 1663, in open', '    raise ReadError("file could not be opened su
ccessfully")', 'tarfile.ReadError: file could not be opened successfully']'

[Jannis Leidel]

This is weird, I can't seem to reproduce this realiably on each run, which I guess means there is something going on with the LiveServerTestCase used for implementing the URL test cases.

[Aymeric Augustin]

This test fails because the live server is unable to serve the /admin_scripts/custom_templates/project_template.tgz; instead, it runs into an infinite redirection loop.

This problem occurs with any existing file, it isn't specific to project_template.tgz.

C:\Documents and Settings\Administrator\Desktop>C:\Python25\python.exe "\\vmware-host\Shared Folders\django-trunk\tests\runtests.py" --settings=tests.test_sqlite admin_scripts.StartProject.test_custom_project_template_from_tarball_by_url
Creating test database for alias 'default'...
Creating test database for alias 'other'...
> \\vmware-host\shared folders\django-trunk\tests\regressiontests\admin_scripts\tests.py(1479)test_custom_project_template_from_tarball_by_url()
-> self.assertTrue(os.path.exists(os.path.join(testproject_dir, 'run.py')))
(Pdb) p args
['startproject', '--template', 'http://localhost:8081/admin_scripts/custom_templates/project_template.tgz', 'urltestproject']
(Pdb) import urllib2; urllib2.urlopen(args[2])
*** HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
FOUND
(Pdb) import urllib2; urllib2.urlopen('http://localhost:8081/admin_scripts/custom_templates/project_template/manage.py')
*** HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
FOUND
(Pdb) import urllib2; urllib2.urlopen('http://localhost:8081/')
*** HTTPError: HTTP Error 404: NOT FOUND

It appears that django.views.static.serve receives a "Windows-style" path instead of an "URL-style"path:

C:\Documents and Settings\Administrator\Desktop>C:\Python25\python.exe "\\vmware-host\Shared Folders\django-trunk\tests\runtests.py" --settings=tests.test_sqlite admin_scripts.StartProject.test_custom_project_template_from_tarball_by_url
Creating test database for alias 'default'...
Creating test database for alias 'other'...
> \\vmware-host\shared folders\django-trunk\django\views\static.py(33)serve()
-> path = posixpath.normpath(urllib.unquote(path))
(Pdb) path
u'\\admin_scripts\\custom_templates\\project_template.tgz'
(Pdb) request.path
u'/admin_scripts/custom_templates/project_template.tgz'
(Pdb) bt
  c:\python25\lib\threading.py(462)__bootstrap()
-> self.__bootstrap_inner()
  c:\python25\lib\threading.py(486)__bootstrap_inner()
-> self.run()
  \\vmware-host\shared folders\django-trunk\django\test\testcases.py(1062)run()
-> self.httpd.serve_forever()
  \\vmware-host\shared folders\django-trunk\django\test\testcases.py(943)serve_forever()
-> self._handle_request_noblock()
  \\vmware-host\shared folders\django-trunk\django\test\testcases.py(982)_handle_request_noblock()
-> self.process_request(request, client_address)
  c:\python25\lib\socketserver.py(241)process_request()
-> self.finish_request(request, client_address)
  c:\python25\lib\socketserver.py(254)finish_request()
-> self.RequestHandlerClass(request, client_address, self)
  \\vmware-host\shared folders\users\myk\documents\dev\django-trunk\django\core\servers\basehttp.py(139)__init__()
  c:\python25\lib\socketserver.py(522)__init__()
-> self.handle()
  c:\python25\lib\wsgiref\simple_server.py(138)handle()
-> handler.run(self.server.get_app())
  c:\python25\lib\wsgiref\handlers.py(92)run()
-> self.result = application(self.environ, self.start_response)
  \\vmware-host\shared folders\users\myk\documents\dev\django-trunk\django\contrib\staticfiles\handlers.py(67)__call__()
  \\vmware-host\shared folders\users\myk\documents\dev\django-trunk\django\contrib\staticfiles\handlers.py(68)__call__()
  \\vmware-host\shared folders\users\myk\documents\dev\django-trunk\django\core\handlers\wsgi.py(241)__call__()
  \\vmware-host\shared folders\users\myk\documents\dev\django-trunk\django\contrib\staticfiles\handlers.py(58)get_response()
  \\vmware-host\shared folders\django-trunk\django\test\testcases.py(1002)serve()
-> document_root=self.get_base_dir())
> \\vmware-host\shared folders\django-trunk\django\views\static.py(33)serve()
-> path = posixpath.normpath(urllib.unquote(path))

The request handler used by the LiveServer is:

handler = StaticFilesHandler(_MediaFilesHandler(WSGIHandler()))

And here is the definition of _MediaFilesHandler:

class _MediaFilesHandler(StaticFilesHandler):
    # ... snip ...
    def serve(self, request):
        return serve(request, self.file_path(request.path),
            document_root=self.get_base_dir())

self.file_path is likely to be the culprit. This code was introduced at r17241.

I believe the solution is simply to replace self.file_path(request.path) by request.path.

At least, it fixes the test failure.

[Aymeric Augustin]

In [17416]:

Fixed #17469 -- Prevented a double URL-to-filesystem path translation in the media handler of LiveServerTestCase, which led to infinite redirection loops under Windows and test failures.

[Aymeric Augustin]

In [17417]:

Improved the fix of r17416 so it actually works on all platforms. Refs #17469.