language agnostic - Pattern for website content that updates periodically? -

I often find myself preparing simple small web projects that are completing composite material or doing 'mashup' Are there. Generally, it involves scraping / parsing some scripts / manipulating from time to time, then serves as a 'stable' material.

I run the 'refresh' script as a cron job, which generates HTML which is served, content for the end user does not change often, so usually I only need one hour I can run a cron job on the basis of.

Is there a better way to do this?

If you are happy that how it is working now, then I will not change anything. It's a havoc, but a functional one. But I'm guessing that you are not completely happy (otherwise you would not have asked) so a more important answer is as follows.

A basic upgrade would be to write a script that generates your mashup sources and generates HTML on-the-fly mashup source, from remote web server to local files, to local databases, anything can happen - Whatever code can be "connected". The basic steps will be:

1. Recover information from each source, programmatically.
2. Parsing it and converting as needed, skip the bits you do not want and maybe redesign some parts like date formats etc.
3. Inject all different bits of your converted information into the HTML conversion and output to the client.

1 & amp; Basically the 2 sounds like you are already doing. This is only # 3 which is the missing link. You basically want to make an out-of-the-fly output rather than dynamically sending it to a predicting and stable HTML.

PHP, Perl, Ruby, Python, and others for things like this; Make your selection.

Further optimization - In order for you to probably do it - include:

1. Caching source data instead of voting sources with each page load , Select them for the first time, save the answer to a file or database, and check to see the timestamp of the reaction on subsequent page loads whether it is still "refreshing" or not. If so, you can send a local cache copy instead, usually improving performance on a larger scale.
2. Is loading its source data asynchronously, which means the time it takes to load becomes the slowest source instead of that source.
3. For example, loading and loading each source immediately through different AJAX calls, displaying each page in their own device.