Concerned about Ditto, PHx etc not working as per in Evo 1.2.1

[nick0]

Hello,

This is just a general comment, not an issue.

At the moment I am hesitant to try the new Evolution because it sounds like some Evo bread and butter things like Ditto, PHx et al are not working properly. Is anyone else feeling the same?

What has changed so much between 1.2.1 and the new Evolution that cause these Evo staples to stop working?

Backwards compatibility needs to be maintained for all Evo users and to encourage uptake of the new Evolution. If standard functionality breaks on upgrades, its going to be a hard sell and a lot of effort to upgrade existing 1.2.1 sites.

[bossloper]

Agreed. First I must say the new features/development time by all involved is greatly appreciated! However these minor/point releases should not compromise compatibility in the name of development.
The new features are great, but I also have many sites still on 1.1.x with a few additional security patches.... because it just works.
It should be possible to upgrade from 1.x.x easily. That way the community will grow and provide more/wider future help and development.

[pmfx]

I haven't have any problems with Ditto on final 1.3.3 release. I had some issues with PHx compatibility though. Had problems with Ditto on dev version (that was scary!) but this seems to be fixed know.

Phx - I miss it's stability. But to be honest built-in filters now looks somewhat stable and does the job for small things. And for bigger ones snippet "if" or custom code should be used to get best performance. Remember that snippets inside phx "then" "else" statements were always executed even if hidden.

Anyway, we should pay more attention and try not to break things like Ditto. Many minor things can have small issues/bugs but not a parser or featured extras like Ditto/eForm that are used by thousands.

I think next release will be stable as 1.3.3 or more.

Kudos for all devs for their hard work!

[Pathologic]

Ditto and other bread and butter are gone bad long time ago. What evolution we are talking about, if you want to keep this obsolete stuff forever?

[pmfx]

@Pathologic thats OK but if Ditto is too old and need to go, then users need to have time to make transition to new solution. Should be notified 6 months ahead that Ditto will not be supported and presented with a alternative (DocLister) with examples. All with use of the official website. To much users depends on those snippets and they are bundled with all releases. All should be clear and easy to understand without "guessing" on whats going on. Communication is most important.

[bossloper]

@Pathologic I would think any element bundled with a release should be backwards compatible. After all by including them with the install that is effectively endorsing those elements as maintained and good for the system.

As @PFMX suggested if a bundled element is to be 'decommissioned' then some sort of warning process should be followed... maybe even an entry in the system log like for old database calls with a link to documentation on how to replace it? Then after a period of time it can be removed from the bundled elements.

Looking forward to the next release!

[nick0]

Thanks for the input everyone. That was a good read and think the majority of us are on the same page.

I LOVE Ditto and am not sure what is obsolete about it. Its so flexible and easy to use.

If there is a new equivalent to Ditto it doesn't really matter as long as it does eveything ditto can do (filtering, sorting) and more (hierachical anyone?). But backwards compatability has to be maintained otheriwse it will be huge job to update our client sites if Ditto calls stopped working.

I am not sure if this is feasible but when ClipperCMS changed Ditto to List, Ditto calls still worked somehow using the updated List code. Some mapping like that would be cool if possible.

Just on the PHx thing, sounds like we are still in the MODx Evo PHx vs Modifiers vs If issues discussed a little previously but that Modifiers has been chosen as the defacto for 1.3+.

Is there an equivalent to this PHX call "phx:striptags" in the Evolution 1.3 modifiers?

// Example usage: [+content:striptags=`<a><img>`+] 
return($modx->stripTags($output, $options));

What about phx:word_limit?

// Usage: [+content:word_limit=`30`+]
// Usage: nested to strip out html [+content:word_limit=`30`:striptags+]
$retval = $output;
$array = explode(" ", $output);
if (count($array)<=$options)
 {
  $retval = $output;
 }
 else
 {
  array_splice($array,$options);
  $retval = implode(" ", $array);
 }
return $retval;

Does this PHX call work as is in the Evolution 1.3 modifiers?
[+pub_date:date=%B %e, %Y+]

And do the modifiers work in MultiTV templates? eg

[+iteration:eq=`1`:then=` first`:else=``+]
[+title:isnot=``:then=` data-title="[+title+]"`

Thanks and keep up the great work.

[modxuser]

My Opinion:

It was said quite a while back that DocLister was/is supposed to replace Ditto, but at the time I asked if DocLister can replicate "EVERYTHING" that Ditto can do - the answer was at that time "NO".

So saying that something that is a default snippet distributed in the main package is old and redundant is in my opinion incorrect - if there is no other solution that replicates all what Ditto can do, it will carry on being used

May I suggest that DocLister be modified to incorporate "ALL" of the things that Ditto can do and the documentation be updated to reflect all of the changes

[bossloper]

... it would also be useful if there was a good overlap period when both Ditto and DocLister are fully working. Then those upgrading are not in a situation of needing to immediately update all Ditto call to get a site back online. Instead having time to learn DocLister and covert the calls without too much stress.

[Pathologic]

@modxuser DocLister has better code and can do everything Ditto does, even much more, to say nothing of DocLister based components such as SimpleGallery or eFilter. In general, we need a clear roadmap to understand what we are developing and to discuss changes before they occur. @Dmi3yy promised to write it.

[pmfx]

While we are on Ditto & DocLister matter. I was testing DocLister today. It's cool and converting basic/semi-complex Ditto calls is pretty straightforward. That's good.

But there is one thing that Ditto does better or maybe easier: Tagging.

In Ditto tags are built-in feature and very easy to use. In DocLister however, it's more complicated. I couldn't find any reasonable examples and documentation for tagging to solve it. Docs at http://www.evolution-docs.com/extras/doclister/ says only that you can use controller called site_content_tags

output Evo documents filtered by tags, in conjunction with the TagSaver plugin

What is TagSaver? Had no clue but I found this http://modx.im/blog/addons/374.html and even after Google Translator its difficult to understand.

Tagging should be as easy as it is in Ditto where you define any TV with tags, delimiter etc. like:

$params['tagData']='documentTags';
$params['tagDelimiter']=',';
$params['tagDisplayDelimiter']=', ';
$params['tagDocumentID']='15';
$params['tplTagLinks']='@CODE:<a href="[+url+]">[+tag+]</a>';

and in item template you have <p>Tags: [+tagLinks+]</p> to display linked tags. So simple.

Can DocLister do it similar way? Maybe I just couldn't find it in the docs?

[Pathologic]

@pmfx TagSaver and site_content_tags controller provide more proper way of handling tags. Storing tags and relations between tags and resources in separate tables is more efficient solution than Ditto offers. But you are right, tagging needs to be documented. I'll take care about it. Probably all DocLister docs need revision to make them more full and clear.

[modxuser]

@Pathologic

Well, sort of.

Looking at the post that @pmfx wrote, that jogged my memory, as that post query was one of the reasons I haven't converted to using DocLister, as the useage is not the same.

I have a few sites with multiple thousand pages where virtualy all of the pages use Ditto Tagging, in one form or another - either basic tagging or using tags for sub-sectioning (Category Tagging).

So converting a site to use Doclister is not an easy solution, especially when you have thousands of resources that have multiple tags.

Please correct me if I am wrong.

[pmfx]

@modxuser I think you're right. Thats why we need fully working and tested Ditto.

@Pathologic I see why tags in separate table are better and thats good. Thanks for working on documentation!

It would be great if we could have in DocLister something like &tagsSource parameter with tv or table options. tv would allow Ditto tags parameters/placeholders logic while table would switch to DocLister current tags defaults (separate mysql table).

[Pathologic]

Probably, it's worth to tell what's wrong with Ditto tagging and filters in general. The way Ditto's working is to get rows from DB and drop some of them with PHP according to &filter value. While DocLister builds SQL-queries to get only needed data and that's why it's more efficient (compare: to get 100 documents and then iterate over them to drop 99 or to get 1 document immediately). The next problem of tagging consists of storing tags in TV as string, for example, "tag1, tag3, tag4". It's easy to output, but how to build query to search resources with "tag3"? WHERE LIKE 'tag3,%' OR LIKE '%,tag3,%' OR LIKE '%,tag3'? Queries with LIKE are slow. That's why TagSaver get tags from tv and saves them to separate tables, so it's still easy to output tags but searching by tags becomes more fast.

So, Ditto's tagging is bad by design and it would be better not to emulate it, but to run simple script to re-save documents with TagSaver plugin enabled 😸

include_once (MODX_BASE_PATH . 'assets/lib/MODxAPI/modResource.php');
$doc = new modResource($modx);
$q = "SELECT `id` FROM {$modx->getFullTableName('site_content')} WHERE `deleted`=0 AND `published`=1";
$ids = $modx->db->getColumn('id', $q);
foreach ($ids as $id) {
    $doc->edit($id)->save(true, false);
}
$doc->clearCache();

By the way, and I thought about it only now, what's the panic about Ditto? Even if it will be removed from the box, it's still possible to install with Extras. Current troubles are not connected with decision to remove Ditto, it's the consequences of DBAPI bug fixing and @yama will solve it.

[nick0]

@Pathologic - its not panic but concern :). For me the concern is potential for broken sites on updating Evolution if Ditto or PHx is not working. Updating is a big enough task as it is without the stress of ditto and phx calls breaking too. That would be a huge headache.

I had a look at the DocLister documentation. Looks good at a glance and will try in due course.

Does anyone know about those modifier equivalents for the PHx snippets posted earlier above?

[bossloper]

@Pathologic - nice explanation- thanks.
We seem to have come full circle from @nick0 original post, with your last point being key.
So if @yama can fix the DBAPI for Ditto we should be good with regards Ditto. Not sure about PHx, could this issues also be related to the DBAPI?.
The current Ditto situation is not a good advertisement for Evo. I feel this should be a priority fix for the next release to allow more of the existing community to upgrade and so provide more real world feedback for 1.3
Then we will be able to look at all the DocLister goodness at our leisure :-)

[Nicola1971]

The current Ditto situation is not nice and we can't tell to people "You should switch to DocLister and rewrite all your Ditto calls", because sounds like "You should switch to Revolution" :)

DocLister can replicate "EVERYTHING" that Ditto can do

I've used DocLister in last 2-3 years for some new projects and is very cool and faster, but I still use Ditto in most of old websites/projects, Why?
Because I' have not found a way to move from Ditto to Doclister without a lot of SEO 404 issues caused by different URL PATH of Tags and paginations (ie: doclister does not use ?start= in pagination and ?tags= in tagging).

I know that DocLister tagging and pagination works better, but is not something that I can explain to the customer that receives hundreds of errors from the Google Search Console

[Dmi3yy]

Yes but i think ditto place in extras and no place in core Who want use can be instal in 2-3 click ;) Отправлено с iPhone

…

21 авг. 2017 г., в 18:32, Nicola ***@***.***> написал(а): The current Ditto situation is not nice and we can't tell to people "You should switch to DocLister and rewrite all your Ditto calls", because sounds like "You should switch to Revolution" :) DocLister can replicate "EVERYTHING" that Ditto can do I've used DocLister in last 2-3 years for some new projects and is very cool and faster, but I still use Ditto in most of old websites/projects, Why? Because I' have not found a way to move from Ditto to Doclister without a lot of SEO 404 issues caused by different URL PATH of Tags and paginations (ie: doclister does not use ?start= in pagination and ?tags= in tagging). I know that DocLister tagging and pagination works better, but is not something that I can explain to the customer that receives hundreds of errors from the Google Search Console — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[pmfx]

@Dmi3yy sure Ditto can be installed from extras (like DocLister) but we need to be 100% sure Ditto works as expected and is compatible with latest and future EVO releases. It is probably one of the most used EVO snippets in history so there is little space for bugs here IMO.

[Dmi3yy]

Yes its need support ) Отправлено с iPhone

…

21 авг. 2017 г., в 20:47, Piotr Matysiak ***@***.***> написал(а): @Dmi3yy sure Ditto can be installed from extras (like DocLister) but we need to be 100% sure Ditto works as expected and is compatible with latest and future EVO releases. It is probably one of the most used EVO snippets in history so there is little space for bugs here IMO. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Nicola1971]

I've just started this reference table for basic Ditto to Doclister parameters conversion, if anyone wants to help and contribute :)
https://github.com/Nicola1971/Ditto-to-DocLister-Reference-Tables

[sashabeep]

+1 PHX not working anymore, Ditto partially

@Pathologic any news about new modifiers syntax in DocLister? I had no clue in my attempts to use it, luckly DL still utilizes PHx syntax for basic conditions

[Jako]

Are you really deprecating the most important core snippet that was used in MODX Evolution without a real migration path? Don't call it 1.3 if you can't maintain backwards compatibility.

[Dmi3yy]

DocLister great alternative for Ditto. And now we need delete Ditto from core.

Ditto is morally obsolete. And also its code is outdated. If you look at the last 20 tasks, then we see that all the compatibility problems in Ditto and in phx

I believe that we must move forward. I understand that learning new is very difficult and no one wants. But the doclister gives much more opportunities and I note that lately there were only questions with him about how to implement this or that functionality, but there were no questions about what's not working there.

Today, DocLister covers the Dittto functionality 100%

[Dmi3yy]

I believe that it is up to the team that supports and develops the EVO to make a decision about what should be in the core. Since we do this with a look at the community, but we have a little more knowledge and understanding. Therefore, if we say that it's time to go off, then it means it is.

EForm and Weblogin have long been out of date. If a great alternative to FormLister. But the community again drags us back because they are used to eForth and do not want to learn new things. And again we are faced with resistance from the community. Instead of asking you to teach and explain what are the new solutions better and how to solve the problem. The community continues to use old solutions.

I understand this is true with the support of old sites. But why use old solutions on new sites?

[Jako]

It is not a learning 'new things' issue. It is a dropping backwards compatibility issue.

And thats not a trivial one. It's a sudden knockout for a lot of old installations that can't get any security updates anymore without changing snippet calls on the page.

[Nicola1971]

@Dmi3yy I agree, but i still think is not the right time for it.
we first need to allow people to update to new Evo 1.x without backwards compatibility issues.
Then, after created a good documentation for DocLister and FormLister, we can move old extras out from the core

[yama]

Sorry during the discussion. I am fixing Ditto. Could you please show me the actual sample?

[pmfx]

And we are talking about Ditto bugs and current/long term compatibility issues. Like I said, Ditto can be moved from bundle to extras, but it have to be tested more and compatible.

@yama
#187
#188
#190

[Dmi3yy]

Yes. For now we need close all bugs in Ditto and only then move to extras

[Nicola1971]

Yes. For now we need close all bugs in Ditto and only then move to extras

I think it's better fix, update and leave Ditto in the 1..3.5 release. So people don't need update ditto from extras.
then remove from the core in 1.3.6

[bossloper]

I agree Nicola.
Yes it makes sense for new projects to learn and use DocLister, but for EXISTING projects there is currently no migration path.
We should not make the same mistake as Revo and abandon existing Modx Evo installs!
To upgrade a 1.1 install without significant downtime I will currently need to duplicate the site, upgrade it and then fix it before copying it live... a big undertaking for many sites,

+1 for 1.3.5 being a 'migration path' release.

[Nicola1971]

We should not make the same mistake as Revo and abandon existing Modx Evo installs!

This Revo mistake is probably one of the main reasons why Evo still exists.
We must not forget and repeat the mistake without a migration path

[Deesen]

A list of migration-instructions would be nice like several Ditto- and Doclister-calls in direct comparison.

[Nicola1971]

A list of migration-instructions would be nice like several Ditto- and Doclister-calls in direct comparison.

@Deesen I'm not a doclister expert, but i've started a parameters comparison.
Working on it, for me, is a good way to learn Doclister :)
https://github.com/Nicola1971/Ditto-to-DocLister-Reference-Tables

any help is appreciated

[fourroses666]

I found great documentation here: http://saniock.com/modx-evolution/extras/doclister.html

At the moment testing a sitemap using Ditto converting to DocLister.
Looks good beside the filters which I need to find out howto.

[Dmi3yy]

For sitemap. better usee DLSitemap )

all docs will be here with multilang.
docs.evo.im

http://docs.evo.im/en/extras/doclister.html

in near future want add comments and search. and new help with translate and add full doc
https://github.com/evolution-cms/docs/

[modxuser]

DocLister English documentation is already in the English Evo Docs, you can find it here

Whether or not it is up-to-date is beyond me, but I should think it's a good start

@Dmi3yy , @Deesen , @pmfx , @Nicola1971 , @fourroses666

My two cent:

Doclister isn't a 100% compatible replacement for Ditto, as has already been mentioned multiple times, specificaly the way it works with tagging, it also uses different URL parameters - meaning, a website with multiple thousands of pages will probably be penalised by Google once it changes from Ditto to Doclister, alone due to the different parameters used in the URLs. Whether or not that would happen is currently open, but if the URL changes, you will have a ton of 404s for the old URLs.

I haven't moved over simply due to this major incompatibility.

Also I would like to raise a point about what you said:

I believe that it is up to the team that supports and develops the EVO to make a decision about what should be in the core

I appreciate that you have taken over leadership of the development of Evo, but without the community Evo wouldn't be where it is today.

I have always opened my mouth where others have buckled, so here goes: you say you believe its up to the people who support and develop Evo when it comes to what can and can't be in the system - well, what about the community ? Have you actualy asked the community their opinion ?

I see loads and loads of modifications being made to the system, yet I don't see anyone in the community being asked if they want the modifications or not. I was until now the opinion that Evo "is" still a community driven project, with the community users in mind, not a project that is now being built around the needs of one or a small group of users.

At the present moment, I think that Evo is going in the wrong direction - backwards compatibility is a big issue that has to be looked at.

I have no problems with "new in Evo", I have problems with no fundemental documentation and or resources that are not comprehensable and easy to use.

If Ditto is to be removed. so be it, but give the community a resource that provides all comparable solutions needed to convert all Dittos calls "1 to 1" to DocLister, including URL parameters

I hope that no-one comes back on this saying "why don't you help", sorry, but I am not a PHPr as all know who know me in the community, but I have helped over many years where I could, with tutorials, the re-build and improvement of the complete English documentation and answering questions in the forum, so I think I have the right to say what I think.

[pmfx]

OK, about Tagging. Maybe it will be easier than I thought.

I was looking for a "Ditto to DocLister tags" solution and came up with using DocLister filters to use TV with tags delimeted by comma, previously used by Ditto. I'm not sure if it is bulletproof but looks like it is working.

You need to add filters parameter in your DocLister call.
New snippet call in DocLister item tpl where tags should appear
Create new snippet to render tags links.

Here are examples:

1. Required params in DocLister call.
   When $_GET "blog_posts_tags" variable have value, filters are used.
   https://gist.github.com/pmfx/bef18541b1835d0855ececf231fa973d

2. DocLister item tpl:
   https://gist.github.com/pmfx/9b2852ecb4a9f5b2d8034db58f46a8d0

3. New snippet to link tags to tags landing page with "blog_posts_tags" $_GET variable set:
   https://gist.github.com/pmfx/66da4628fbfd34d7a7d4019d70287c07

[modxuser]

I am about to go to bed, but have just seen your post, so:

Hi @pmfx, is that solution compatible with all of the tagging methods, as seen here:

http://www.evolution-docs.com/tutorials/extras-how-to/ditto-tutorials/category-tagging-with-ditto
http://www.evolution-docs.com/tutorials/extras-how-to/ditto-tutorials/tagged-blogging-with-ditto-ex-1
http://www.evolution-docs.com/tutorials/extras-how-to/ditto-tutorials/tagged-blogging-with-ditto-ex-2

And what about the future of TagLinks, TvTagCloud and other snippets that work together with Ditto ?

I just looked at point 3 in your list - that wont work with category tagging, as each category has its own ID - or am I wrong here ?

[pmfx]

Tagging methods (&tagMode) - not sure. Probably yes with different &filters operator.
TagLinks, TvTagCloud and other - probably not.

No doubt we need to have Ditto fixed, but this was just a quick research about how to use TVs based tags with DocLister.

[nick0]

I agree wholeheartedly with the sentiments above about maintaining backward compatibility and community involvement / consultation. I am all for progress and improvements and new ways of doing things for new projects, but the new Evolution has to work for existing MODx Evo 1.2.1 websites. Its not progress if the sites break on install. It's evident that many people are really passionate about Evo. Lets keep it that way.

[Nicola1971]

Is there a doclister placeholder to show the author of the page in tpl? like Ditto [+author+] placeholder?

[Pathologic]

&extender=`user` &usertype=`mgr` &userFields=`createdby,publishedby` &tpl=`@CODE:[+user.username.createdby+] [+user.fullname.publishedby+]`

[Nicola1971]

Thank you @Pathologic. very nice
I have tried the user extender but not the userFields parameter :)

[Pathologic]

&usertype=`web` &userFields=`tv.aid` &tpl=`@CODE:

[+pagetitle+] ([+user.fullname.tv.aid+])

` - it works with TVs as well

[Nicola1971]

very cool, need to learn more about doclister user extender

I'm tring to convert an old ditto site to doclister to see any possible issue and the time required for the port.

About pagination:

It would be nice to have a &paginateQuery param in Doclister to customize the ?page= and fit old Ditto
?start= pagination and avoid 404 errors

[Pathologic]

there's ditto-like pagination mode - &paginate=`offset`, though it's not developed because of no need.

[Nicola1971]

but seems working! it's a good news :)

[yama]

[[Ditto?tpl='[+editedby:username+]<br />']]
[[Ditto?tpl='[+pub_date:dateformat(Y-m-d)+]<br />']]

Related information

[Nicola1971]

@pmfx just tested your solution for tags and added some modifications to the snippet.
Now i can replace ditto tags using the same url structure, and works with TvTagCloud snippet :)