0

Starzel.de Python Blog

 3 years ago
source link: https://www.starzel.de/blog/starzel-de-python-blog
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

Starzel.de Python Blog

Python 3, Plone 5.2, Zope 4 and more

erstellt von Philip Bauer —

zuletzt geändert: 2018-10-19T10:14:08+01:00

Report from the SaltLabs Sprint in Halle, Germany and the aftermath

With over 30 attendees the SaltLabs Sprint was among the bigger sprints in the history of Plone and Zope.

Fortunately, our host Gocept just moved into a new office and had plenty of space to spare. Stylish as we are, the Plone crowd quickly congregated in the palace-like salon with red walls and renaissance paintings. Even better: Throughout the week the cafe Kaffej downstairs was open for us to eat breakfast, lunch and for work. We tried to keep the barista busy producing excellent Cappuccino. Thanks!

Zope 4

The sprint split in two groups, one working on Zope and one working on Plone. A short report on about what happened at the Zope can be found at https://blog.gocept.com/2018/10/11/beta-permission-for-earl-zope-extended . Zope 4.0b6 was released just after the sprint. The new Beta 6 includes the new bootstrap-styled ZMI, support for Python 3.7 and a metric ton of bug fixes compared to 4.0b5 which was released in May this year. By the way: Zope can now be installed without buildout using pipenv

Python 3

The Plone crowd was huge and worked on several tasks. The biggest task - and the main reason to join the sprint - was continuing to port Plone to Python 3. Jens Klein, Peter Mathis, Alessandro Pisa, David Glick (remote) and Philip Bauer as lead worked mainly on fixing all remaining failing tests and getting the jenkins-builds to pass. We dove deep into crazy test-isolation-issues but also added a couple of missing features to Dexterity that were so far only tested with Archetypes.

During the sprint we got very close to a green build but a couple of really nasty test-isolation issues prevented a green build. A week after the sprint we finally got that done and all tests of Plone now pass in Python 2.7, Python 3.6 and Python 3.7 with the same code-base!

With the test passing we're now merging that work and preparing the release of Plone 5.2a1. The tasks for this are discussed in this ticket

Deprecate Archetypes

Archetypes will not be ported to Python 3 but still has to work when running Plone 5.2 in Python 2.7. Since all tests that used PloneTestCase were still using Archetypes Philip changed that to use Dexterity now and created a new testlayer (plone.app.testing.bbb_at.PloneTestCase) that is now used by all the packages using the Archetypes-stack. See https://github.com/plone/plone.app.testing/pull/51. You could use this layer in your archetypes-based addon-packages that you want to port to Plone 5.2.

Addons

Yes, Plone is now able to run on Python 3. But without the huge ecosystem of addons that would be of limited real-world use. Jan Mevissen, Franco Pellegrini (remote) and Philip Bauer ported a couple of addons to Python 3 that Plone developers use all the time:

Being able to run tests against Plone in Python 3, and with these development-tools now available also, it is now viable to start porting other addons to Python 3. To find out how hard that is I started porting collective.easyform. The work is only 97% finished since some tests need more work but it is now useable in Python 3!

ZODB Migration

Harald Friesenegger and Robert Buchholz worked on defining a default migration-story for existing databases using zodbupdate. To discuss the details of this approach we had a hangout with Jim Fulton, David Glick and Sylvain Viollon. They solved a couple of tricky issues and wrote enough documentation. This approach seems to work well enough and the documentation points out some caveats but the ZODB migration will require some more work.

Relevant links:

ZODB Issues

Sune Broendum Woeller worked on a very nasty and complex issue with KeyErrors raising while releasing resources of a connection when closing the ZODB. That happened pretty frequently in our test-suites but was so far unexplained. He analyzed the code and finally was able to add a failing test to prove his theory. Then Jim Fulton realized the problem and wrote a fix for it. This will allow us to update to the newest ZODB-version once it is released. See https://github.com/zopefoundation/ZODB/issues/208 for details.

WSGI-Setup

Thomas Schorr added the zconsole module to Zope for running scripts and an interactive mode. Using WSGI works in Python 2 and Python 3 and will replace ZServer in Python 3.

Frontend and Theming

Thomas Massman, Fred van Dijk, Johannes Raggam and Maik Derstappen looked into various front-end issues, mainly with the Barceloneta theme. They closed some obsolete tickets and fixed a couple of bugs.

They also fixed some structural issues within the Barceloneta theme. The generated HTML markup now has the correct order of the content columns (main, then left portlets, then right portlets) which allows better styling for mobile devices. Also in the footer area we are now able to add more portlets to generate a nice looking doormat. See the ticket https://github.com/plone/plonetheme.barceloneta/pull/163 for screenshots and details.

They also discussed a possible enhancement of the Diazo based theming experience by including some functionality of spirit.plone.theming into plone.app.theming and cleaning up the Theming Control Panel. A PLIP will follow for that.

New navigation with dropdown support

Peter Holzer (remote) continued to work on the new nativation with dropdown-support which was started on collective.navigation by Johannes Raggam. Due to the new exclude_from_nav index and optimized data structures the new navigation is also much faster than other tree based navigation solutions (10x-30x faster based on some quick tests).

Static Resource Refactoring

Johannes Raggam finished work on his PLIP to restructure static resources. With this we no longer carry 60MB of static resources in the CMFPlone repository, it allows to use different versions of mockup in Plone, enable us to release our libraries on npm and will make it easier to switch to a different framework. Skin scripts removal Katja Süss, Thomas Lotze, Maurits van Rees and Manuel Reinhardt worked hard at removing the remaining python_scripts. The work is still ongoing and it would be great to get rid of the last of these before a final Plone 5.2 release. See https://github.com/plone/Products.CMFPlone/issues/1801 for details. Katja also worked on finally removing the old resource registry (for js and css).

Test parallelization

Joni Orponen made a lot of progress on speeding up our test-runs by running different test-layers in parallel. The plan is to get them from 30-60 minutes (depending on server and test-setup) to less than 10 minutes. For a regularly updated status of this work see https://community.plone.org/t/ci-run-speedups-for-buildout-coredev/6225

Documentation and User-Testing

Paul Roeland fought with robot-tests creating the screenshots for our documentation. And won. A upgrade guide to Plone 5.2 and Python 3 was started by several people. Jörg Zell did some user-testing of Plone on Python 3 and documented some errors that need to be triaged.

Translations

Jörg Zell worked on the german translations for Plone and nearly got to 100%. After the sprint Katja Süß did an overall review of the german translation and found some wording issues with need for a discussion

Plone React

Rob Gietema and Roel Bruggink mostly worked on their trainings for React and Plone-react, now renamed to “Volto”. Both will be giving these trainings at Plone Conference in Tokyo. On the second day of the sprint Rob demoed the current state of the new react-based frontend for Plone.

Assorted highlights

  • In a commit from 2016 an invisible whitespace was added to the doctests of plone.api. That now broke our test in very obscure ways. Alessandro used some dark magic to search and destroy.
  • The __repr__ for persistent objects changed breaking a lot of doctests. We still have to figure out how to deal with that. See https://github.com/zopefoundation/zope.site/issues/8 for details.
  • There was an elaborate setup to control the port during robot-tests. By not setting a port at all the OS actually takes care of this to makes sure the ports do not conflict. See https://github.com/plone/plone.app.robotframework/pull/86. Less is sometimes more.
  • Better late than never: We now have a method safe_nativestring in Products.CMFPlone.utils besides our all-time favorites safe_unicode and safe_encode. It transforms to string (which is bytes in Python 2 and text in Python 3). By the way: There is also zope.schema.NativeString and zope.schema.NativeStringLine.
  • We celebrated the 17th birthday of Plone with a barbecue and a generous helping of drinks.

Porting Plone to Python 3

erstellt von Philip Bauer —

zuletzt geändert: 2018-05-24T09:40:11+01:00

&tldr; Plone is running nicely on Python 3 but now we have to fix over 500 failing tests.

Since I wrote the proposal to Port Plone to Python 3, so much has happened that a status update is needed.

Let's step back a little: The first steps towards Python 3 were taken during the sprint at the Plone conference in Barcelona. The epic PLIP to update to Zope 4 was merged and we started porting individual packages to Python 3 without being able to run any tests. With the help of sixer and python-modernize we tried to get the most obvious import and syntax issues out of the way. That work was continued at the Alpine City Sprint in Innsbruck and in the end, we treated more than 150 packages like this. Some of this work is documented in the Plone tracker.

Along the way I wrote a best-practice guide on how to port Plone packages this way.

As the PLIP states, there are several approaches to porting Plone to Python 3:

  1. Migrate and test packages that have no dependency to CMFPlone and test them in Python 2 and 3.
  2. Prepare packages for Python 3 without being able to test them in Python 3.
  3. Start up Plone on Python 3 and fix whatever breaks. When start-up works, create a Plone Site and again, fix whatever breaks.
  4. Port plone.testing and plone.app.testing to Python 3 and start running tests. Fix what breaks during the setup of the layers.
  5. Run the tests with Python 3 and fix all broken tests.

At the sprint in Innbruck I started with #3 and kept going after the sprint until I was able to create an instance. At the Plone Tagung in Berlin I was able to demo the creation of a site but nothing was rendered yet.

After that, I kept going and I was finally able to create a site and manage content, which is what a CMS is about. It looked a bit raw but I was able to add and edit some content - yay!

early screenshot of plone with python 3

Work continued at an unsteady pace, and with important contributions from Michael Howitz, Alessandro Pisa and David Glick, things started to get better. Even the theme and js started working. Slowly broken features became the exception, not the rule.

Last week at the Zope 4 Welcome Sprint in Halle we removed all feature blockers that we had found so far. Plone with Python 3 looks and feels like it used to in Python 2. During the sprint there was also a lot of progress on:

  • the wsgi setup
  • logging and tracebacks when using wsgi
  • porting plone.testing
  • a new theme for the ZMI
  • beta releases for many packages.

There was also some progress on the difficult issue of database migrations. It seems like zodbupdate is the best tool to do that but there is probably a lot of work ahead.

Read more about the sprints outcome in the blogpost by Michael Howitz.

There is a Jenkins job and as of today, it is running (not passing) tests for all packages (except Archetypes) with Python 3.

At the moment we run 6594 tests from 115 packages with 257 failures and 315 errors - not bad when you keep in mind that we still focus on features, not on tests. Tests for a couple of packages are green, including plone.api which is great since plone.api uses most of our stack one way or another. Jenkins runs every three hours and here you can see the progress over time.

Here is a screenshot from today that shows Plone doing some pretty non-trivial things: Editing a recurring Event and also Richtext with Images and Links:

screenshot today

Next steps:

  • Create a demo site running on Python 3 that people can use to find broken features. This will happen at the Plonator Sprint in Munich.
  • Fix all tests that look like they fail because of a broken feature.
  • Fix all remaining tests to pass in Python 2 and Python 3. Since there are a gazillion doctests that will take some time.
  • Port plone.app.robotframework to python 3 and fix robottests.
  • Experiment with porting a ZODB with Plone to Python 3.

If you want to participate you can simply look at the failing tests and start fixing them. You can also try to fix one of the open issues.
The setup of a coredev environment with Python 3 is really simple and documented.

Hello Thomas!

erstellt von Philip Bauer —

zuletzt geändert: 2017-03-02T15:56:48+01:00

We're excited to welcome Thomas Lotze to the team of Starzel.de.

Thomas has many years of experience developing complex web projects with Python, especially using Pyramid and Plone. His interests also include automated testing and questions of software infrastructure. He shares our dedication to Open Source and we are looking forward to many exciting projects in the future.

Aside from development, Thomas does trainings on subjects that he knows well from day-to-day work, in particular Python, Pyramid and pytest.

You can meet us next at World Plone Day and PyConWEB in Munich.

Towards Plone 6

erstellt von Philip Bauer —

zuletzt geändert: 2017-03-01T11:52:13+01:00

A report from the Alpine City Sprint in Innsbruck

Two of the main items on the Plone roadmap are the move to Python 3 and a more active role in the development of the underlying stack of Plone, namely Zope. After the recent uptake of activity in development of Zope (see here and here) it was a sensible first task to try to get Plone to run on the most recent Zope stack. What exactly does that mean?

There is a PLIP that proposes to upgrade Plone's Zope-related dependencies and is approval by the Plone Framework Team to be a part of Plone 6. As with all PLIPs, the point is not only to agree on a task and a implementation but also actually doing the work. After two smaller efforts at sprints last year where only 2 or 3 people were working on the task we finally got a critical mass of developers together for the Alpine City Sprint. 15 people sprinted and worked on this one task.

essen.jpg

Off to a good start

On the day before the sprint I rebased some of our branches and with a couple of quick and really dirty fixes I got Plone run on Zope 4. Many features were still not working but a working instance certainly provided motivation for this very dedicated sprint.

Bildschirmfoto 2017-02-13 um 11.43.41.png

WSGI replaces the ZServer

The ZServer is now a optional dependency and the new WSGI-Publisher is the default publisher making Plone fully WSGI-compliant! Thomas Schorr added some necessary buildout-parts to run Plone on uwsgi, waitress and gunicorn. Documentation is here and examples are here. Even the testbrowser used for all of Plone's functional tests now uses the WSGI-Publisher. The advantages of WSGI are obvious: It makes it easier to have interact Plone with other services and some features can be developed as WSGI-middleware. It'll be interesting to see if we will make more use of WSGI for Plone in the future than "simply" for serving.

ZCatalog 4 and ZODB 5

The result of the sprint is beyond what was originally proposed in the PLIP. We now not only use the master branch from Zope and the zope-toolkit packages (zope.interface, zope.component etc.) but also ZODB 5 and ZCatalog 4.

One change that required some work was that ZCatalog, as it now returns a empty list when you pass no query or a invalid query. Especially in the cases when you query the catalog for a non-existing index (portal_typos='foo') getting all brains from the Catalog seems like a bad idea. The down-side that change there is that there is no obvious way to walk through all brains in the catalog any more. Johannes Raggam added the convenience-method catalog_get_all for this.

The changes in ZODB are numerous but for Plone-developers nothing much really changes except that is supposed to be faster and supports Python >= 3.3 and 2.7. For the future it will be also interesting to discuss if switching to NewtDB would be a option. NewtDB runs ZODB 5 with RelStorage and stores data as json in Postgres.

Bildschirmfoto 2017-02-13 um 11.12.27.png

Python 3

The sprint did not try to move Plone to Python 3 yet but rather clear the way so we can do it. The last remaining blocker with C-code was RestrictedPython and Alexander Loechel and Michael Howitz continued working on the rewrite based on the Python AST module. At the end of the sprint they were happy to announce that it is no longer a blocker for our move to Python 3. All the other packages that rely on C-code that seems mysterious to mere Python-mortals (ExtensionClass, AccessControl, Acquisition, Persistence) were already ported to Python 3. RestrictedPython is a very useful tool that allows you to execute untrusted Python code that is loaded within a running Python program. It checks and modifies source code to only allow the execution of a restricted subset of Python.

What remains to be done?

  • Fix some failing tests (DONE): As of today we are down to 8 failing tests 6 failing tests 2 failing tests 0 failing tests!!! 
  • Update tasks and tickets
  • Merge changes: A lot of the changes were already merged in the master-branches for the current coredev (for Plone 5.1). But some merges will have to wait until Plone 5.1 is released because they only target Plone 6.0. Once all test are green and all our zope-related branches are merged we can make a release of Zope 4.0a3 and switch from using source-checkouts for everything (auto-checkout = *) to using released versions.
  • Documentation: We need to document what changed in Zope from the perspective of Plone. Most notably some lesser features of the ZMI are gone. These and other changes need to go into the upgrade-guide for the next Plone-Version. Also the new setup for various WSGI-Servers needs to be well-documented so that developers can  copy&paste best-practice configs into their buildouts. The documentation was started in a Google Doc where everyone can contribute. Once it's stable we'll move it to their respective places.
  • Upgrades: So far it seems like the changes we made did not make any upgrade-steps necessary. After adding a quick hack to work around the missing zope-controlpanel I was able to upgrade a existing Plone 5.0.6 site without any issues.
  • Profiling: Initial results with from plone.instanceprofiling indicate the speed of Plone has increased but we need more real-life tests before a final judgement. It will also be interesting to see the impact of Python 3 on speed.
Bildschirmfoto 2017-02-13 um 11.15.54.png

Inside the sprint

As usual during the sprint we ran into a bunch of blockers and hard nuts to crack. The hardest one was probably that the current users roles were mysteriously dropped during a request after running trough some C-code. Like the memory of a bad dream, the bug vanished in the light of the morning - actually someone switched to different branch of AccessControl. On the first day already, Matthew Wilkes had committed to a metaclass in Persistence (thankfully brown-bagging my terrible late-night hack and replacing it with something much more elegant), and installed gdb to debug the issue with the dropped roles.

Unfortunately many of our tests were broken since the testlayer did not use chameleon, thankfully after a lot of debugging the solution found by Thomas Schorr was pretty simple. Also tough was a issue that broke tests in weird ways. Gladly Jens Klein, our host, dug deep and found the culprit: Combining bundles modified the response's Content-Type header, surprise! A issue similar to https://github.com/plone/Products.CMFPlone/pull/1438.

Bildschirmfoto 2017-02-13 um 11.22.11.png

Shops!

By the way: We also had a demo of bda.plone.shop and a discussion about shop-solutions for Plone. bda.plone.shop has plenty of great features and seems really well thought-through. It still lacks some end-user documentation with screenshots and a new release but we will certainly use it in our next project that involves a shop.

Where do we go from here?

To the pool in Sorrento! Plone Open Garden (PLOG) in beautiful Sorrento is the perfect opportunity to discussing the further roadmap to Python 3 and continue working on this. PLOG will have a strong focus on the headless CMS project for which a Python 3-compatible Plone 6 will be a mayor milestone. Register now and enjoy a week at the pool with the Plone community.

2016-03-26 18.03.01.jpg

And then on to Finnland to the Midsummer Sprint! All the work on the backend needs to be balanced with some fun UX and UI work to enhance the content editors experience.

Summary

We worked on many important parts of the Plone Roadmap: Removing unused components, clean up the code base, moving to Python 3, and making Plone WSGI compliant. There is now a good chance that Plone will be on Python 3 this or next year. A great sprint, dedicated and smart people, good beer and food and nice mountains. Thanks to the organizers of Klein und Partner who worked hard to keep us fed, fueled and entertained and thanks to all the sprinters who dedicated so much of their time for such a great open source project!

Update (19.02.2017)

  1. Last week gocept announced there will be another Zope sprint in Mai with the goal to port Zope to Python 3. Already 11 developers signed up.
  2. David Glick fixed the remaining failing tests (mosty issues with the new handling of exceptions).

Push coverage report out of an Gitlab CI Runner

erstellt von Steffen Lindner —

zuletzt geändert: 2017-02-06T17:26:17+01:00

For our Plone/Python projects we often generate coverage reports as HTML sites, this posts show how you can push this report out of the Gitlab CI Runner.

 Save FTP Password & Login as secret variables in Gitlab > Settings > Variables. They can be accessed in the .gitlab-ci.yml as $FTPLOGIN & $FTPPASSWORD.

 

gitlab_variables.png

Here is our .coveragerc file, which sets the output directory:

 [report]
include =
    src/plone.site/*
omit =
    */test*
    */upgrades/*
    */browser/dummy.py
[html]
directory = parts/test/htmlreport
title = Plone Project

And our .gitlab-ci.yml:

before_script:
    - /usr/bin/virtualenv .
    - ./bootstrap_ci.sh
job1:
  script:
     - export DISPLAY=:99
     - ./bin/code-analysis
     - ./bin/coverage erase
     - ./bin/coverage run -p --source=src bin/test || exit 1
     - ./bin/coverage combine
     - ./bin/coverage html
     - ./bin/coverage report
        # Stage robot
     - ./bin/test --all -t 'robot' || exit 1
     - cd parts/test/htmlreport/ && find . -type f -exec curl --ftp-create-dirs -T {} -u $FTPLOGIN:$FTPPASSWORD ftp://45.54.45.54/uhlmann/$CI_BUILD_ID/{} \;
  artifacts:
        expire_in: 1 week
        paths:
            - parts/test/

The interesting part is the curl one, it uploads all coverage files to a FTP server.

If your static webserver is public accessible use basic auth with a .htaccess file:

htpasswd -c .htpasswd starzel

.htaccess file:

AuthUserFile /var/www/htdocs/.htpasswd
AuthGroupFile /dev/null
AuthName "Please Enter Password"
AuthType Basic
Require valid-user

Better integration into Gitlab coming soon as Gitlab Pages see this Pull Request.

Todo:

  - Cron to delete old directories.

  - Upload files if robot / code-analysis failed.

Magic templates in Plone 5

erstellt von Philip Bauer —

zuletzt geändert: 2017-03-01T12:22:05+01:00

Due to the new rendering-engine chameleon it is fun again to write templates

Plone 5 uses Chameleon a its rendering engine. Did you know that because of that you can put a pdb in a template? If you saw the keynote by Eric Steele on Plone 5 you probably do.

But did you also know that the variable econtext holds all current variables up to the moment the pdb is thrown?

Let's put a pdb in newsitem.pt:

(...)

<metal:content-core fill-slot="content-core">
    <metal:block define-macro="content-core"
          tal:define="templateId template/getId;
                      scale_func context/@@images;
                      scaled_image python: getattr(context.aq_explicit, 'image', False) and scale_func.scale('image', scale='mini')">

<?python import pdb; pdb.set_trace() ?>

<figure class="newsImageContainer"
         tal:condition="python: scaled_image">
        <a href="#"
           tal:define="here_url context/@@plone_context_state/object_url;
                       large_image python: scale_func.scale('image', scale='large');"
           tal:attributes="href large_image/url">
          <img tal:replace="structure python: scaled_image.tag(css_class='newsImage')" />

(...)

When rendering a News Item the variable scaled_image is accessible as econtext['scaled_image']:

> /Users/philip/workspace/test/f56e9585b89e34318d1171acc17f531a7a428a1f.py(132)render_content_core()
(Pdb) econtext['scaled_image']
<plone.namedfile.scaling.ImageScale object at 0x110073b90>
(Pdb) econtext['scaled_image'].width
200 

You can inspect the whole econtext:

(Pdb) from pprint import pprint as pp
(Pdb) pp econtext
{'__convert': <function translate at 0x10fb4d7d0>,
 '__decode': <function decode at 0x10fb4d578>,
 '__slot_content_core': deque([]),
 '__slot_javascript_head_slot': deque([]),
 '__translate': <function translate at 0x10fb4d7d0>,
 'ajax_include_head': None,
 'ajax_load': False,
 'args': (),
 'body_class': 'template-newsitem_view portaltype-news-item site-Plone section-super-news userrole-manager userrole-authenticated userrole-owner plone-toolbar-left-default',
 'checkPermission': <bound method MembershipTool.checkPermission of <MembershipTool at /Plone/portal_membership used for /Plone/super-news>>,
 'container': <NewsItem at /Plone/super-news>,
 'context': <NewsItem at /Plone/super-news>,
 'context_state': <Products.Five.metaclass.ContextState object at 0x10db00910>,
 'default': <object object at 0x100291bf0>,
 'dummy': None,
 'here': <NewsItem at /Plone/super-news>,
 'isRTL': False,
 'lang': 'de',
 'loop': {},
 'modules': <Products.PageTemplates.ZRPythonExpr._SecureModuleImporter instance at 0x102e31b48>,
 'nothing': None,
 'options': {},
 'plone_layout': <Products.Five.metaclass.LayoutPolicy object at 0x10db00310>,
 'plone_view': <Products.Five.metaclass.Plone object at 0x10db00dd0>,
 'portal_state': <Products.Five.metaclass.PortalState object at 0x10fbe8d50>,
 'portal_url': 'http://localhost:8080/Plone',
 'repeat': {},
 'request': <HTTPRequest, URL=http://localhost:8080/Plone/super-news/newsitem_view>,
 'root': <Application at >,
 'scale_func': <Products.Five.metaclass.ImageScaling object at 0x10c2bf390>,
 'scaled_image': <plone.namedfile.scaling.ImageScale object at 0x110073b90>,
 'site_properties': <SimpleItemWithProperties at /Plone/portal_properties/site_properties>,
 'sl': False,
 'sr': False,
 'target_language': None,
 'template': <Products.Five.browser.pagetemplatefile.ViewPageTemplateFile object at 0x10fff3ed0>,
 'templateId': 'newsitem.pt',
 'toolbar_class': 'pat-toolbar initialized plone-toolbar-left',
 'translate': <function translate at 0x10fb4d7d0>,
 'traverse_subpath': [],
 'user': <PropertiedUser 'adminstarzel'>,
 'view': <Products.Five.metaclass.SimpleViewClass from /Users/philip/workspace/test/src-mrd/plone.app.contenttypes/plone/app/contenttypes/browser/templates/newsitem.pt object at 0x10cd93910>,
 'views': <Products.Five.browser.pagetemplatefile.ViewMapper object at 0x10f5cc190>,
 'wrapped_repeat': <Products.PageTemplates.Expressions.SafeMapping object at 0x10ffba1b0>}

Using n you can actually walk down the template and inspect new variables as they appear. After pressing n about 11 times the variable large_image appears as econtext['large_image'].

(Pdb) econtext['large_image'].width
768

The pdb-session you are in is no restricted python but real python. This means you can do the following:

(Pdb) from plone import api
(Pdb) portal = api.portal.get_tool('portal_memberdata')
(Pdb) memberdata = api.portal.get_tool('portal_memberdata')
(Pdb) memberdata.getProperty('wysiwyg_editor')
'kupu'

Hey, what is kupu doing there? I found that in some of our sites that were migrated from Plone 3 this old setting prevented TinyMCE to work in static portlets. But that is a different story, let's just get rid of it.

(Pdb) memberdata.wysiwyg_editor = 'TinyMCE'
(Pdb) import transaction; transaction.commit()

This is a full-grown pdb and you can inspect and modify your complete site with it.

But there is more: You can actually have complete code-blocks into templates:

<?python

from plone import api
catalog = api.portal.get_tool('portal_catalog')
results = []
for brain in catalog(portal_type='Folder'):
    results.append(brain.getURL())

?>

<ul>
    <li tal:repeat="result results">
      ${result}
    </li>
</ul>

Quick and dirty? Maybe dirty but really quick! It is still very true that having logic in templates is bad practice but I think in some use-cases it is ok:

  • Debugging
  • When you customize a existing template (with z3c.jbot or plone.app.themingplugins) and need some more logic
  • When you quickly need to add some logic to a browser-view that only has a template but no class of it's own

Have fun templating with Plone 5! If you want to learn more about Plone 5 you can still register for the training "Mastering Plone 5 Development" in March 2.-6. (http://www.starzel.de/leistungen/training/)

Update:

As disucced here you can add the econtext to the the locals() by using the following stanza:

<?python locals().update(econtext); import pdb; pdb.set_trace() ?>

Plone Conference 2014: The Highlights

erstellt von Philip Bauer —

zuletzt geändert: 2017-03-01T12:22:06+01:00

The Plone Conference has once again proven its value. There were many excellent talks and everyone had a great time. In Open Spaces and during many discussions between talks the current state and the future of Plone became much clearer.

Plone 5

The main thing is Plone 5. It look great, is a huge step forward in terms of user experience and brings tons of important improvements. The keynote of Plone's release manager Eric Steele showcases all these things.

Getting Plone 5 ready for a beta-release is the top item on everyone’s agenda. That effort not only involves finishing projects such as mockup, the new theme and contenttypes but also writing documentation for end-users, the upgrade-guide for developers and testing.

The Plone Roadmap

The second biggest thing was the roadmap-discussion. Discussions about the a new Roadmap started in October 2013 at the Plone Conference in Brazil under the catchy title "Plone Roadmap 2020". The initial topic was the future of Zope within Plone's ecosystem, but it turned into a broader discussion about all the things that the community wants to change in Plone, not forgetting the whys and hows.

Roadmap progress in Bristol 2014

First we looked at reasons why we use Plone. The following motivations were mentioned the most:

  • The Plone Community
  • Fully-featured user friendly product that is directly usable
  • Flexibility of the software architecture
  • It creates jobs
  • Best security-record
  • Open source

As Martin Aspelli put it so eloquently: If we are going to change Plone, can we please not mess any of the above up?

Next we compiled a list of things that we want to change in Plone in the next five years.

  • python-api
  • json-api
  • Improve end user experience
  • Documentation and training: Document the recommended way to do things
  • Improve TTW-story in theming, templating and customization
  • Simplify code base, reduce number of eggs, remove legacy technologies
  • Remove dependency on CMF
  • Roadmap communication

Not a single person advocated a rewrite since it is clear that a rewrite would mess with some things on the first list.

When looking closer at the aims it becomes clear that all of these things are already being worked on:

  • python-api: There is now a Plone Improvement Proposal to ship plone.api with Plone and also use it in the core. There will be further discussions about what we need from an api to further isolate Plone from parts that we might want to replace in the future.
  • json-api: The community is already working on a RESTful json-api for Plone that will allow isolating the front-end from the backend and experiment with various javascript-frameworks. With this api Plone will be better suited for for web-applications where Plone is "only" used as the backend and will be a friendly citizen in a world of mixed technologies.
  • UX: Plone 5 will already be a huge step in the right direction but there will always be room for improvement. The enthusiastic reaction to the efforts of the Plone Intranet Consortium showed how important a good UX is.
  • Documentation: The new docs.plone.org are much better than anything Plone had in the past. During the sprint the documentation was already being cleaned of examples that use grok. Also the documentation for the Mastering Plone Training teaches always the recommended ways to do things and will be expanded and upgraded for Plone 5.
  • TTW-Story: Although Mosaic (the new name for Deco) looks extremely promising this is surely one of the areas where Plone still needs to invest more. For template-customization we still rely on old technologies: The only way to customize a viewlet within Plone is to use the ZMI (not recommended!). The community will have to agree on achievable solutions to have some real progress. But do not forget that Plone already has a great TTW-story: Dexterity and the Diazo theme-editor are powerful features and blow the competition out of the water.
  • Simplify code base, reduce number of eggs: Many technologies (formlib, portal_skins, plone-tools, cpy/cpt) are already deprecated and the code is migrated to browserviews and z3c.form (*cough*) in Plone 5. There is also a PLIP that aims to move many plone.app-packages into the core-package Products.CMFPlone [https://dev.plone.org/ticket/13283].
  • Remove dependency on CMF: Without a python-api we cannot remove parts of CMFCore and Zope. Having that api will give us the options we need to replace/remove stuff. We'll need some experiments and even more discussions about what we want to remove and what to replace it with.
  • Roadmap communication: The discussion on a the Plone Roadmap will be continued. The next Plone Open Garden (April 2015 in Sorrento) will probably be turned into a Plone Strategic Planning Summit. Communicating the community's strategic vision of Plone in the short, mid and long term to the public is almost as important as agreeing on a roadmap and implementing it.

Since there is no official roadmap-team these kind of meetings are where the future of Plone is actually agreed on. This is where Plone happens and everyone who takes part is part of the roadmap-team.

The Plone Intranet Consortium

One of the most anticipated talks of the conference was about the Plone Intranet Consortium. Since its foundation last year, the consortium (of which Starzel.de is a founding member) is working on a competitive Social Intranet solution. One of the aims is to evolve and strengthen the position of Plone in a commoditized market. The design-first approach and the fact that Plone-companies put their resources together allows for high expectations. See the talk:

The sprint

As usual after the conference there were two days to actually work on Plone. The hotel had to quickly open another big room to accommodate all sprinters since many more people showed up this time than  expected. On saturday morning you could see the titanpad exploding when people started adding the topics they wanted to work on. I worked with a great team of developers on the default contenttypes of Plone 5, mainly focusing on making it easy to migrate the content of existing websites to Dexterity. Other results can be seen at the titanpad and here.

In 2015 the Plone Conference will be in Bucharest. I wouldn't miss it for the world.

658eb094-f6f8-4a39-b15f-25eb3679b7c3.jpeg

Sprinting towards Plone 5

erstellt von Philip Bauer —

zuletzt geändert: 2017-03-01T12:22:06+01:00

The drive towards Plone 5 is gaining momentum as the goal seems to be in a reachable distance.

For the first time there actually is a state of Plone 5 that is can be shown to people who are interested in the future of Plone. The essential building-blocks of Plone 5 are starting to look presentable: The new barceloneta-theme is really nice and comes with the new toolbar and the new widgets.

Don't get me wrong, there is still quite some work to be done before a final release but the first alpha-release is in the making and the current state can actually be shown. These screenshots are from a fresh build of the coredev.buildout.

plone5-left-2.png plone5-top-2.png plone5-edit-2.png

To reach Plone 5 the Plone-community is organizing a staggering number of sprints. My sprint-plan for 2014 looks like this: Cologne, Munich, Munich (again), Sorrento, Berlin, Bristol.

I recently attended the cathedral-sprint in Cologne, organized by the tireless Timo Stollenwerk and generously sponsored by GFU Cyrus AG. Timo, whose sole purpose in life sometimes seems to be to keep Jenkins happy, organized another impressive display of the spirit of the Plone-community as 30 people from around the world gathered there to work in groups on pushing Plone 5 towards readiness. It is great to see so many originally separate efforts coming together smoothly and making a whole new thing that will be at the core of Plone 5.

Together with Johannes Raggam, Wolfgang Thomas and some help from others I was continuing the work on plone.app.contenttypes and we made good progress. The current version 1.1b2 of the new default-content-types for Plone 5 has a multitute of bugfixes and some relevant changes compared to version 1.0:

  • Collections are now a behavior that can be enabled on any content-type. The collection-type is basically a folder with that behavior.
  • The shiny new event-type now uses the behaviors provided by plone.app.event including support for recurring events and timezones.
  • Richtext-fields are now a behavior shared by most types.
  • We now have migrations from 14 different content types to the new types including migrations for four (!) iterations of the event-type.

I have been using plone.app.contenttypes since before the project officially existed and I think every plone-developer should give it a try and report any issues especially regarding migrations to make it safe to ship with Plone 5.

It was a real pleasure being reunited with old friends and making new ones. It is especially encouraging that even at sprints there are always new people showing up and contributing to Plone.

The next big sprint on my agenda is the Wine-and-Beer-Sprint in Munich. For the second time we are organizing this sprint together with our friends from Syslab.com and again it is a double-sprint with a leg in Munich and one in Capetown. We will continue working towards Plone 5. The necessary tasks to get Plone 5 out of the door are summarized in this ticket: https://github.com/plone/Products.CMFPlone/issues/184.

Register now since we only have a limited amount of places! If you really need additional incentive to attend, here they come: A fine selection of Single Malt Whiskeys, a great sprint-location that is accessible 24h, Sichuan Hot Pot and white sausages (not to be mixed).

The Wine and Beer-sprint is is preceded by the DocSprint (also in Munich) that focuses on consolidating, improving and versioning(!) the documentation for Plone.

See you in Munich and elsewhere!

Pydev Extensions now Open Source!

erstellt von Philip Bauer —

zuletzt geändert: 2014-06-30T13:03:12+01:00

A great tool for python (and Zope/Plone)-development just got better

The good News:

Pydev is a great eclipse-Plugin for which I primarily use for Plone- and Zope-Development. Pydev 1.5 now includes the formerly closed-source Pydev Extensions. Check out it's amazing features.

Learn more about Plone-Development with Eclipse and Pydev at my talk at the dzug-conference in Munich next week.

The bad news:

It happend only two weeks after I renewed my License for Pydev-Extensions :-(

Developing Plone on Ubuntu 9.04 with virtualenv

erstellt von bleicher —

zuletzt geändert: 2011-02-18T15:52:24+01:00

Technical notes to overcome some obstacles

Ubuntu 9.04 "Jaunty Jackalope" removed python 2.4 libraries from the distribution, which are necessary to run Zope/Plone.

To work in a more defined environment, where other as the system libraries are installed for a specific python, people use virtualenv.

Install virtualenv, python-2.4 from the distribution repository.

Create a 

virtualenv -p python2.4 --no-site-packages myappdir

In myappdir,

source bin/active

sets some environment variables.

I use zsh with the "nounset"-Option, which the script doesn't work with, so I had to

unsetopt nounset

Install ipython or zc.buildout or other eggs you need

easy-install ipython

PIL aka python-imaging is not longer available in a a 2.4 version in Ubuntu 9.04, but contains a lot of binary code so I didn't want to build it myself.  3 options: Install an old python-imaging-2.4-package,

Install it by hand - easy_install --find-links http://download.zope.org/distribution PILwoTK or untar and python setup.py install and before that, you have to install some other image handling libs to get all the features as explained here  http://sam.stainsby.id.au/blog/?p=28

sudo apt-get install libjpeg-dev libfreetype6-dev (zlib?)

or. what I did, install the current package and link to it from the virtualenv.

In myappdir/lib/site-packages

ln -s /usr/lib/python2.5/site-packages/PIL PIL

Python complains about differing C API versions, but it seems to work.

This all looks less than perfect, esp. the PIL link hack.

Too many packaging tools.

Functional Python

Die objektorientierte, dynamisch getypte Programmiersprache python, in der der web application server Zope und das content management system Plone geschrieben sind, entwickelt sich stetig auf eine überzeugende Art weiter.

Ein Beispiel dafür ist der elegante Einbau von funktionalen Sprachmerkmalen wie unendlichen Listen (als Generatoren und Iteratoren) in python.

# Sieb des Erastothenes
from itertools import count
from itertools import ifilter
def sieve(possible_primes):
  next_prime=possible_primes.next()
  yield next_prime
  for i in sieve(ifilter(lambda j: j%next_prime!=0, possible_primes)):
    yield i

def primes():
  return sieve(count(2))

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK