How to search complex source code using regular expressions

Today I needed to search the source code that ware calling a function with a complex array of parameters

URL::Make( 'site/store.inc.php', array( 'action'=>'view','id'=> [... STH...] ,'idTwo'=> [... STH...] ) );

Considering this requirements:

- there are lots of similar function (to exclude from the search) with an argument less or an argument more

- there may are additional spaces in the line

- some other delopers have probably used double quotes instead of single quotes for array keys

- the value of the keys may be a PHP code

- every function call is in one line

An elegant solution is to search using regular expression. Netbeans (my favorite editor) supports POSIX extended regular expression and is very quickly to search a complex epression in a huge amount of source code.

Solution 1

#using char length and patter ".{0,MAX}" (that is: every sequence, MAX maximum chars). Easy !

URL::Make.{0,10}site/store.{0,40}action.{0,9}view.{0,20}id.{0,200}idTwo.{0,30}\).{0,10}\)

Solution 2

#using regexpr for separators. more complex. more accurate in some cases but fails in some cases (eg: comments inside)

URL::Make.*\([ ]*['"]site/store.inc.php['"][ ]*,[ ]*array[ ]*\([ ]*['"]action['"][ ]*=>[ ]*['"]view['"][ ]*,[ ]*['"]id['"][ ]*=>.{0,100}['"]idTwo['"][ ]*=>.{0,100}[ ]*\)[ ]*\)

An idea for DB server load balancing

If the application has a heavy load of reading queries and there are no problems with the application requirements,

a simple possible idea to balance the db server loading is to run the INSERT/UPDATE/ALTER queries in all the servers and run the SELECT queries in just one server (chosen randomly).

Have a look at the 'loadBalancedQuery' function: if the query modifies the db it will be executed in both server, otherwise in just one.

class BalancedDB extends Mysqli {

private function queryServer($query, $server = "mainServer" ) {
$db = DB::GetInstance($server);
return $db->query();
}

public function loadBalancedQuery($query) {
if (preg_match("/[]*(insert|INSERT|update|UPDATE|alter|ALTER).*/",
$query)){
$this->queryServer( $query, "mainServer" );
$this->queryServer( $query, "secondaryServer" );

} else {
$this->queryServer( $query, rand(0,1) ? "mainServer": "secondaryServer" );
}
}

}

UTF8 in LAMP applications: overview and how to solve the common issues

The problem

In a LAMP application the text is frequently saved/retrieved from/to a database and files. We must consider all the different encoding (mapping characher-byte value): latin1 iso8859-1, latin9, UTF-8 (utf8) etc...

Lots of applications use ISO8859 encoding and some PHP functions to convert the characters (htmlentities, htmlspecialchars etc...)

The solution

Converting all the text from an encoding to another using PHP functions is unsafe, difficult and annoying.

Ad example the character "é" (encoded as iso8859) will printed as "é" if it's supposed to be encoded as utf8.

The solution is to use only one charset for files, Content-Type of the pages and the db. UTF-8 [wiki] is the best choice: a variable-lenght char encoding for the standard Unicode. If you use this charset in the HTML file, it won't need to convert the characters to the respective entities.

How To use UTF-8

• set your IDE to save and open source files using utf8 encoding
• set the content-type of your application to utf-8 (better a apache/htaccess rule instead of the meta tag).
• set the database server to use utf8 encoding (also tables must be converted). If the db is utf8 but client encoding is latin1, execute first of all the query "SET NAMES utf8"
• if the application was using latin1 and PHP convert functions, remove all the existing function to encode/decode special characters/entities.

Mass rows copying (duplicating) with filed customization - MySQL

Let's suppose we have a table with the following structure and data:

# `table`
id | a | b | c |
---------------------------
1 | "aaa" | "xxx" | "ccc"
2 | "aab" | "yyy" | "ccc"
3 | "aba" | "yyy" | "ccc"
4 | "abc" | "xxx" | "ccc"
5 | "dcz" | "xxx" | "eee"


Now, we want to copy some records (only some columns) to the same table AND change some of them with a fixed value.
Requirements:
- duplicate all the rows with `b` equal to "xxx"
- for the new rows inserted, the value of `c` has to be changed to "ddd" (fixed value!)
- `id` is the primary key, auto increment

Query (pay attention to the quote characters):
INSERT INTO `table`
(SELECT

NULL as `id`,

`a`,

`b`,

"ddd" as `c`

FROM `table`

WHERE `b`="xxx")

Result:

id | a | b | c |
---------------------------
1 | "aaa" | "xxx" | "ccc" #row1
2 | "aab" | "yyy" | "ccc"
3 | "aba" | "yyy" | "ccc"
4 | "abc" | "xxx" | "ccc" #row4
5 | "dcz" | "xxx" | "eee" #row5
6 | "aaa" | "xxx" | "ddd" #copied from #row1
7 | "aab" | "xxx" | "ddd" #copied from #row4
8 | "dcz" | "xxx" | "ddd" #copied from #row5

Note 1:

The insert operation by using a subquery is not an 'atomic' operation for the recent versions of MySQL. If there are table constraints and a new record is not accepted (e.g: duplicate record for a key defined on columns), only that record won't inserted (not all of them!).
Example: If before there was a record with the same values as in line 9 and there was a unique key on columns (a,b,c), the query would insert only rows on line 6,7 and 8 (9 fails).
Some old version of MySQL stop execution in case of duplicate entries. In this case, add IGNORE to the query to skip duplicates: INSERT IGNORE INTO ...

Note 2:

If you add a new column to the table, the query will fail !!

Tested on MySQL 5.1.36

How to recursively check syntax of PHP files

The executable of PHP supports the '-l' option, that checks the syntax instead of parsing the file.
Using the command 'find', it's possibile to do a interesting operation: syntax checking of all the files recursively, to avoid parse errors in some script !!

find ./ -type f -name \*.php -exec php -l {} \; ";

the result will be a list of files, example:

No syntax errors detected in ./codebase/controller/competition.inc.php
No syntax errors detected in ./codebase/controller/feed_data.inc.php
Errors parsing ./codebase/controller/site/contact.inc.php
No syntax errors detected in ./codebase/controller/compare_prices.inc.php

We can improve the script and print only the file with suntax errors using 'grep'

find ./ -type f -name \*.php -exec php -l {} \; | grep "Errors parsing ";

To launch it from a PHP script

passthru('find ./ -type f -name \*.php -exec php -l {} \; | grep "Errors parsing " ');

Updated: To skip .svn directories add the option :
-not -regex '.*/.svn/*.*'

help of useful commands

In this post, just a link of files contains help guide ([command] --help) for the most common unix shell commands.

cp
find
grep
gunzip
gzip
help
ls
mv
ps
tar
wget

Monitor linux web-servers with monit

"monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.

Monit is controlled via an easy to configure control file based on a free-format, token-oriented syntax. Monit logs to syslog or to its own log file and notifies you about error conditions via customizable alert messages. Monit can perform various TCP/IP network checks, protocol checks and can utilize SSL for such checks. Monit provides a http(s) interface and you may use a browser to access the monit program."[ref] .

How to install in ubuntu 8.10

• sudo apt-get install monit
• edit /etc/monit/monitrc and remove comments
  #launch system mon every 2min
  set daemon 120
  #set web interface on port 2812
  set httpd port 2812 and
  # allow localhost and request from IP 1.2.3.4
  allow localhost
  allow 1.2.3.4
  #request username (admin) and password(monit)
  allow admin:monit
  #check apache
  check process apache with pidfile /usr/local/apache/logs/httpd.pid
  #check mysql service (other server)
  check host myserver with address 192.168.1.1
  if failed icmp type echo count 3 with timeout 3 seconds then alert
  if failed port 3306 protocol mysql with timeout 15 seconds then alert
  #set alerts via email
  set mailserver mail.domain.com
  set alert administrator@domain.com
• check the syntax of the previous file with
  sudo monit -t
• edit /etc/default/monit
  startup=1
• start deamon !!
  sudo /etc/init.d/monit start
• call web interface
  http://yourserverip:2812/
  use admin and password specified before
  

Compare Files between two directories recursively - PHP script

In this post I'll present my PHP class to show differences (file missing or different file date) between files in two directories (recursively).
It's useful to compare two websites (Example, test and live versioning the same server) and show the differences, much faster than a FTP synchronization, and customizable (size differences instead of time, PHP copy commands with backup etc...).

Example of output:


How to use:
= new CompareDirs("e:/siti/symfapp/", "e:/siti/symfapp2/");
$c->addSkipPath(".svn".DIRECTORY_SEPARATOR);
$c->addSkipPath(".project".DIRECTORY_SEPARATOR);
$c->addSkipPath(".settings".DIRECTORY_SEPARATOR);
$c->compare();
echo "{$c->totalFiles} files. {$c->skippedFiles} skipped.
{$c->differencesNotFound} missing.{$c->differencesTime} time differences. ".$c->getHTML();
?>


View the code
comparedirs.php

benchmark: boolean variables value check

$a = NULL;

$t = microtime(true);
for($i = 0; $i<10000000; $i++) {
if ($a) {} }
print microtime(true)-$t."\n"; //2.84 sec

$t = microtime(true);
for($i = 0; $i<10000000; $i++) {
if ($a==true) {} }
print microtime(true)-$t."\n"; //3.52sec

$t = microtime(true);
for($i = 0; $i<10000000; $i++) {
if ($a===true) {} }
print microtime(true)-$t."\n"; //3.0 sec


$t = microtime(true);
for($i = 0; $i<10000000; $i++) {
if (isset($a)) {} }
print microtime(true)-$t; //3.08 sec

benchmark: array_key_exists vs isset

When the aim is to check if the element exists in an array and has a value not null, better to use isset(array[index]), that is much faster than array_key_exists(index, array).

#PHP 5.3.0 command line - Amd turion 64 1.6 Ghz

#create array
$a = array();
for($i = 0; $i<100000; $i++) {
$a[] = "$i asd a".($i*2)." sd";
}

#isset
$t = microtime(true);
for($i = 0; $i<10000000; $i++) {
isset($a[$i]);
}
print microtime(true)-$t; //4.16 sec

#array_key_exists
$t = microtime(true);
for($i = 0; $i<10000000; $i++) {
array_key_exists($i, $a);
}
print microtime(true)-$t; //11.7 sec

note: array_key_exists(index, array)and isset(array[index]) are a little different:

$a = array(0=>0, 1=>NULL);
print isset($a[1])?1:0; //0
print array_key_exists(1, $a)?1:0; //1

note: eAccelerator ( and probably also other PHP accelerators) don't improve the array access time

note: xDebug (and probably zend optimizer) activated can slow down a lot the performance (by 6/7 times) with huge arrays like these !

Functional testing for web applications: Selenium IDE for firefox

In a previous post I wrote about PHP Unit testing.

In software science, the levels above Unit testing are Integration, System and System integration testing.

For a standard web application I suggest to use only Selenium [official site], a powerful complete framework , distributed with apache licence 2.0 (a free-software, GPL compatible).

Selenium includes many components. In my opinion the most useful is Selenium IDE [page], a firefox plugin to create action tests, play and check results.

It's very easy to use the IDE [official manual], but to write good tests it needs some practice.

Basically, a test case is a sequence of commands, and a test suite is a cluster of test cases.

To create actions the tool allows to record user actions (clicks, page opening...) or manually insert/edit (mainly to insert verify-actions) and play the test case (or test suite), then check results.

It's possible to export the test cases in many formats (html table, o export (html table format, PHPunit selenium extension, etc...) and to import (only html table).

Every command has only one or two arguments. The command reference is present in selenium IDE. Here are the commands I mainly use and I suggest:

• open dir/page.html
  open the url specified
• type id=phone 123456
  fill in field with id=phone with the text "123456"
• clickAndWait id=elementID
  click on the link with ID equal to "elementID" .
  
• clickAndWait link=text
  click on the link with "text" as internal text (a "text-dependent" approach is plainly not recommended)
• verifyTextPresent done
  Verify the page contains the text "done". If not, the test will fail at runtime. (be careful: "text-dependent").
• verifyElementPresent id=elementID
  Verify the page contains the element with ID equal to "elementID". If not, the test will fail at runtime.
• assertConfirmation
  assert a javascript confirmation appear

Instead of use the ID to locate an element (example: "id=linkA") it's possibile to use XPath expressions [syntax by w3c school] as locators. Example: "//div//a[2]" is the second link inside the "div" tag. I don't suggest to use XPath: if you (or the designer) change the template (position, or text inside links, or change teh text) the test will not work !

To write solid reliable tests, remember to use a univocal ID for elements to test (message confirmation, links, form fields..), and write the test as soon as the application is working (you will easily and quickly test the correct application behaviour after adding new features).

To test CRUD forms, I suggest to write the html code (template) using a different ID for confirmation messages and error messages

[?php if ($result ): ?]

[div id="postok"]submit ok[/div]

[?php else: ?]

[div id="posterr"][?php echo $messageError ?][/div]

[?php endif; ?]

To test the result of this form:

verifyElementPresent postok

or

verifyTextNotPresent posterr

Selenium IDE screenshot

How to solve SVN conflicts. Netbeans SVN plugin tools

SVN [official site] is the most powerful open source version control software at the moment.

Introduction (for who doesn't know it at all)

Basically, SVN is a server/client system. The server that contains the repository (source code of our application).The clients get the source code (CHECK OUT) or only the new files (UPDATE) from the server. The opposite operation is called COMMIT, and consists in sending new files to the server. Every commit generates a new repository revision; SVN mantains the content of the files in every revision with the aim not to lose code and the possibility to undo code changes.

[http://telin.ugent.be/~slippens/svn-intro/fig/vc-concepts.png]

How to solve conflicts

SVN works allowing clients to get files without any locks ! It means that sometimes you may commit a file, but another client has committed the same files before. This is a conflict ! SVN requests to you to manually resolve the conflict, in order not to lose the code written by the other client (correct behaviour !).

How can you solve the conflict ? There are many solutions:

1. Delete your local changes (REVERT). your code will be overwritten by the newest file in the repository (written not by you). Do it if you are sure you want to delete your changes.
2. Keep ur changes and commit. Do UPDATE and SVN client will be overwrite the file (with your code between "[[[[[[[[[[[ .mine" and "=======", and the code in the repository between the second marker and "]]]]]]]]]]]]] .rN"). SVN client will also create a copy of your changed file (file.ext.mine), the original you worked with (file.ext.rM) and the newest file in the repository(file.ext.rN).
   
   now:
   2.1 - Analyze differences and create a new file (recommended)
   analyze the differences (DIFF) between files, then modify the file.txt, set the conflict resolved (RESOLVE, after that SVN client allows to commit) and COMMIT.
   - 2.1.1 - For small files, you can edit directly file.txt (mind deleting the markers) and obtain a new file (operation called merging). Then set resolved and commit.
   - 2.1.2 - For bigger files, it's a good solution to use a tool for file comparison (DIFF), for example between file.ext.mine (your modified file) and file.ext.rN (the newest in the repository). Then save the new file as file.ext, set resolved (RESOLVE) and COMMIT.
   
   or
   
   2.2 - Overwrite the current file in the repository with your changes (be careful !). To do it, copy file.ext.mine to file.ext, set resolved(RESOLVE) and COMMIT

Netbeans SVN plugin

Netbeans [official site] (free java-based IDE for JAVA, J2SE, C/C++, Ruby and PHP) works with the collab.net SVN command line client and provides a useful tools to manage the SVN operation on the source code.

How to do the operations above with Netbeans 6.7 (a message error dialog appears in case of conflict). Filename: file.ext. All the commands are available with right click on the file.ext, then "Subversion".

1. "Revert modification..." and "Revert Local Changes"
2. "Update" (the file will be coloured "red", it' necessary before 2.1.1 or 2.1.2), then:
   
   2.1.1 (imho the best solution). Manually merge the changes and create a new content for the "red" file (mind removing markers). "Resolve Conflicts" (now the colour will be blue, and the temporary files will be deleted), then "Commit".
   
   2.1.2 (Netbeans solution, not the best solution imho) ."Show changes", double click on the bottom panel "Subversion" on the line with the "red" file (status=local conflict). In the main window, you will see the local file (left) and the remote (right). Netbeans will show line by line the conflicts (different code in the same line), and you can choose the line to have in the new file by clicking in "Accept" on the left or on the right. then "Commit".
   
   Netbeans/manual alternative to 2.1.2: Select file.ext.mine and file.ext.rN (where N is the highest), then tight click, "Tools" and "Diff". Now create a copy with the rigth code, then copy all the content in file.ext, "Resolve Conflicts" and "Commit". It Looks complicated but maybe it's the best in case of lots of conflicts.
   
   2.2. copy the content of the hidden file file.ext.mine to file.ext (the red one), then "Resolve Conflicts" and "Commit" (strongly not recommended !!).

How to reduce conflicts

1. Do "Update" before start changing a file !! You will reduce the possibility of conflicts
2. Do more "commit", for example every 30min instead of 2h (choose it in according on the number of the clients working on the same file).
3. In case of merging you will probably have to contact other developers to be sure about merging in case your code to commit is much different. Arrange on files with other developers (you'll avoid to have lots of conflicts when committing)
4. Other suggestions ?

post moved here: solve SVN conflicts netbeans

SVN to FTP/web on Windows with svn client+winscp

Post moved to the new blog: Svn to ftp

How to deploy a web application and to quickly deploy future updates/patches from a SVN repository into a remote web server (only with ftp access and without possibility to setup svn client) ?
Here are my suggested steps:

1. export SVN (that is obtain the last repository without ".svn" directories)
2. process the exported SVN (e.g. adding or deleting files before uploading)
3. upload (1st time) or synchronize (update new files to the remote webserver)
4. chmod folders (only 1st time or new folders)

All the 4 steps are executable from a handy shell script using:

• Linux: svn client (1) + rsync (3)
• Windows: svn client subversion (1) + winscp script (2)

Example of a svn to ftp batch in Windows (with subversion client and winscp, both are free):

1. Create a profile with winscp, and set local folder "c:\temp\site1" and remote folder (e.g. "/httpdocs" ).
2. Write a batch file svn2ftpsite1.bat
   set PATHFOLDER=c:\temp\site1
   svn export svn://1.2.3.4/trunk %PATHFOLDER% --force
   del %PATHFOLDER%\file-not-to-upload\* /q
   copy c:\other-folder\file-to-add-to-site1 %PATHFOLDER%
   c:\path-to\winscp.exe /console "open winscp-profile" "synchronize remote" "chmod 777 uploads" "close"
3. Save. And launch when you want to update the remote site (don't forget to commit the svn before)

Read the winscp online scripting manual for further information.

Note: if the web server (Linux I think!) can access the SVN repository, has SSH server and subversion client, you can export directly the repository in the web server.

Note: if the web application works correctly, take note of the current repository version. in case of future problems you will be able to export that revision (svn client command line option) and restore the online web site.

Note: If you can install svn client in the web server and the repository is visible from the web server, you can use "check out" and "update" directly in the webserver, then use an apache rule to hide .svn directories

RewriteRule /\.svn /errorpage

Unit testing: a brief introduction to PHPUnit

Every software needs testing [wiki], starting with testing software units [wiki] (the smallest parts of a software, that are functions and classes).

Briefly, a test is a code which runs and ensure that our code behaves as expected.

Here is an example of a test to check a mail validator function:

print ( validateEmail("test@site.com") && !validateEmail("test@site") && !validateEmail("@site") && ! validateEmail("with space@site.com") && !validateEmail("test@site.commm") && validateEmail("a-b_c@sub.site.com")) ? "function looks OK":"test fail";

Obliviously, we need more testing features (first of all to understand which test fails). So, let's use an existent tool to make and to run tests effectively and easily.

PHPUnit is a free powerful easy-to-use framework to manage PHP Unit tests.

• Installation: setup with pear (recommended) or download the source archive
• Test writing: include the main class, extend the PHPUnit_Framework_TestCase class and write inside one method for each unit test. Each method should call an existent superclass method, such as assertEquals($a, $b), assertTrue($a), etc... [online manual]
• Run: use a command line tool and type
  phpunit fileWithTests.php
  PHPunit will automatically run all the tests (=methods written in the class) and will show the detailed results of each one.

This is only a summary, consult the PHPunit official online manual to explore all the features.

One personal suggestion: write the tests BEFORE writing the code ! You will save time and write a better code.

PHP 5.3.0 released

I don't like writing posts for every new PHP release as other PHP blogs, but this time it's worth doing it !

PHP 5.3 implements important new OOP features:

• namespaces
• late static binding and __callStatic() !
• anonymous functions
• Phar extension (pratically, it's the same as JAR for java files)

Forother improvments and bug fixing, see change log