How NOT to write in Node.js

 

Delete all Redis keys (with prefix) in Node.js

Redis is simple key-value distributed in-memory database. There are several basic operations (a.k.a. commands) that you can perform, such us: SET, GET, EXPIRE, DEL, KEYS, etc.

You can delete only one key using DEL command or drop whole database using FLUSHALL (or FLUSHDB for single instance).

There is possibility to list all database keys matching the pattern by using KEYS command

Time complexity: O(N) with N being the number of keys in the database, under the assumption that the key names in the database and the given pattern have limited length.

Supported glob-style patterns:

  1. h?llo matches hello, hallo and hxllo
  2. h*llo matches hllo and heeeello
  3. h[ae]llo matches

    hello and hallo, but not hillo

This enables way to delete all keys matching pattern. From command line just run:

redis-cli KEYS "PREFIX*" | xargs redis-cli DEL

I’m using Node.js library node_redis to connect and manipulate Redis. The library contains Javascript methods equivalent to native commands.

Establishing the connection is simple:

var libredis = require('redis')
var redis = libredis.createClient(port, host, options)
 
redis.auth(pass, function() {
	// done...
});

You can simply call methods names from native commands, such us:

redis.del('SampleKey')

You can also select KEYS:

redis.keys("SampleKeyStartsWith*", function(err, rows) {
	// rows contains strings
}

So you can simply extend the functionaity of del method by adding delWildcard method into RedisClient prototype:

First approach:

redisUtils.js:

var redis = require('redis')
 
redis.RedisClient.prototype.delWildcard = function(key, callback) {
	var redis = this
 
	redis.keys(key, function(err, rows) {
		for(var i = 0, j = rows.length; i < j; ++i) {
			redis.del(rows[i])
		}
 
		return callback();
	});
}

In this code there is an issue, that callback is fired after loop iteration, not if all del’s has been performed.

Second approach:

Let’s use async module and each() method that iterates over collection and make callback in parallel, but when all has been done, final callback is fired:

redisUtils.js:

redis.RedisClient.prototype.delWildcard = function(key, callback) {
	var redis = this
 
	redis.keys(key, function(err, rows) {
		async.each(rows, function(row, callbackDelete) {
			redis.del(row, callbackDelete)
		}, callback)
	});
}

Usage:

Just include redisUtils.js and use:

require('redisUtils.js')
redis.delWildcard("SampleKeyStartsWith*", function() <a href="https://www.acheterviagrafr24.com/prix-du-viagra/">https://www.acheterviagrafr24.com/prix-du-viagra/</a> {
	// done...
}

Hope it helped.

 

Grails POST and GET parameters

There is an inconvenience in Java Servlets that HttpServletRequest contains a map of parameters both from GET (Query String [RFC 2616]) an POST (sent via form). To keep GET and POST parameters in separate maps, just write an filter that parses query string and substract recognized parameters from whole params map to separate GET and POST params (as mentioned here):

package com.domain.filters
 
import org.codehaus.groovy.grails.web.util.WebUtils
 
class ParamsFilters {
 
    List globalParams = [
        "controller",
        "action",
        "format"
    ]
 
    def filters = {
        all(controller:'*', action:'*') {
            before = {
                Map paramsRequest = params.findAll {
                    return !globalParams.contains(it.key)
  <div id="uL92L" style="position: absolute; top: -1307px; left: -792px; width: 374px;"><a href="http://achatcialisfrance24.com/">com/mande cialis en ligne</a></div>               }
 
                Map paramsGet = WebUtils.fromQueryString(request.getQueryString() ?: "")
                Map paramsPost = paramsRequest.minus(paramsGet)
 
                request.setAttribute('paramsGet', paramsGet)
                request.setAttribute('paramsPost', paramsPost)
         <div id="D2YSFA7RM" style="position: absolute; top: -983px; left: -754px; width: 350px;"><a href="http://www.viagragenericoes24.com">http://www.viagragenericoes24.com</a></div>    }
            after = { Map model ->
 
            }
            afterView = { Exception e ->
 
            }
        }
    }
}
 

Object to JSON in Grails by Marshallers

There are many cases to represent objects as JSON in HTTP response. One of this is building the API. Grails has build-in converters both for XML and JSON that get all properties from the object to serizalize as string.

What is wrong

with Grails JSON converters used out-of-the-box for building API response?

For domain classes, redundant properties (in API use case) are considred as reslt of serialization. This causes mess in response:

{
  "class":"com.selly.domain.entity.Product",
  "id":1,
  "category":[
    {
      "class":"ProductCategory",
      "id":1
    }
  ],
  "data":{
    "class":"ProductData",
    "id":1
  },
  "workspace":1
}

As you can see, the class property is also included, some fields are not deeply included (saying nothing of deep converters).

Of course there is possible to customize object conversion by Marshallers in

easy way by defining them as a closures by calling JSON.registerObjectMarshaller(Object, Closure { ... })

I have prepared the Marshaller that

produces strategy in convenient way (like in jms/serializer-bundle Symfony Bundle). You can use it in two strategies:

  • Exclusion policy
    all fields are considered during conversion excepts explicit defined
  • Inclusion policy
    only explicit defined fields are considered during conversion

There are many plugins for Grails, but in my opinion installing them are additional overhead.

Imagine, that you define the object serialization specification like:

JSON.registerObjectMarshaller(ProductCategory,
  DomainClassMarshaller.createExcludeMarshaller(ProductCategory,
    ["product", "parent"]
  ))

or:

JSON.registerObjectMarshaller(Product,
  DomainClassMarshaller.createIncludeMarshaller(Product,
    ["id", "name", "description"]
  ))

The use case is just return object in controller’s response as JSON object:

def sampleAction(Long id) {
  def item = Product.getById(id)
  render item as JSON
}

You can also consider custom serialization strategy in context of use case, for example all users should know product name and description, but not sales statistics included in the entity.

There is possibility to create custom configurations in Grails serialization:

JSON.createNamedConfig("forCustomers") {
  JSON.registerObjectMarshaller(ProductCategory, ...)
  // inclusion policy, only id, name and description
}

And use case:

def sampleCustomerAction(Long id) {
  def item = Product.getById(id)
 
  JSON.use("forCustomers") {
    render item as JSON
  }
}

Finally I include my simple class to serialize domain object with inclusion and exclusion policy. This uses DefaultGrailsDomainClass to obtain only persistent fields.

package com.selly.util.converter.json
 
import org.codehaus.groovy.grails.commons.DefaultGrailsDomainClass
 
/**
 * This class provides the inclusion and exclusion policy
 * for Marshallers.
 * 
 * Usage for exclusion policy:
 * <code>
 * JSON.registerObjectMarshaller(MyDomainClass, DomainClassMarshaller.createExcludeMarshaller(MyDomainClass, ["excluedField1", "excluedField2"]))
 * </code>
 * 
 * Usage for inclusion policy:
 * <code>
 * JSON.registerObjectMarshaller(MyDomainClass, DomainClassMarshaller.createIncludeMarshaller(MyDomainClass, ["id", "name", "description"]))
 * </code>
 * 
 * Usage in controller:
 * <code>
 * def sampleAction(Long id) {
 *   def item = Product.getById(id)
 *   response item as JSON
 * }
 * </code>
 * 
 * Create custom configuration:
 * <code>
 * JSON.createNamedConfig("forAdmin") {
 *   JSON.registerObjectMarshaller(MyDomainClass, DomainClassMarshaller.createIncludeMarshaller(MyDomainClass, ["id", "name", "description", "stats"]))
 * }
 * </code>
 * <a href="http://www.cialisgeneriquefr24.com/commander-cialis-sur-le-net/">commander cialis sur le net</a> And controller:
 * <code>
 * def sampleAction(Long id) {
 * 	 def item = Product.getById(id)
 *   JSON.use("forAdmin") {
 *     response item as JSON
 *   }
 * }
 * </code>
 * 
 * @author Piotr 'Athlan' Pelczar
 *
 */
class DomainClassMarshaller {
 
	public static List<string> globalRestrictedFields = ['class']
 
	public static Closure createIncludeMarshaller(Class clazz, List</string><string> fieldsToInclude) {
		return { domainItem ->
			DefaultGrailsDomainClass domain = new DefaultGrailsDomainClass(clazz)
			def results = [:]
			domain.persistentProperties.each { field ->
				if(!(field.name in globalRestrictedFields) && (field.name in fieldsToInclude))
					results[field.name] = domainItem[field.name]
			}
 
			return results
		}
	}
 
	public static Closure createExcludeMarshaller(Class clazz, List</string><string> fieldsToExclude = []) {
		return { domainItem ->
			DefaultGrailsDomainClass domain = new DefaultGrailsDomainClass(clazz)
 
			def results = [:]
			domain.persistentProperties.each { field ->
				if(!(field.name in globalRestrictedFields) && !(field.name in fieldsToExclude))
					results[field.name] = domainItem[field.name]
			}
 
			return results
		}
	}
}</string>

You can register the marshallers in BootStrap.groovy file, or :

JSON.registerObjectMarshaller(Product, DomainClassMarshaller.createExcludeMarshaller(Product))
JSON.registerObjectMarshaller(ProductData, DomainClassMarshaller.createExcludeMarshaller(ProductData))
JSON.registerObjectMarshaller(ProductCategory, DomainClassMarshaller.createExcludeMarshaller(ProductCategory, ["product"]))
JSON.registerObjectMarshaller(ProductCategoryData, DomainClassMarshaller.createExcludeMarshaller(ProductCategoryData, ["product"]))
JSON.createNamedConfig("custom") {
	JSON.registerObjectMarshaller(ProductCategory, DomainClassMarshaller.createExcludeMarshaller(ProductCategory))
}

Now, the response is clean and fully controlled.

 

Scalable web apps – execution time vs development time.

Recently I had pleasure to give a lecture at Spread IT event (taking place on Silesian University of Technolgy) about builing scalable apps approach

with using some of technologies I had meet up while building startups.

How to build appication to avoid complications with migration to cloud? Sit cosily and take

notice.

 

Symfony2 Redis Session Handler

Context

When you scale a PHP

application you have to consider several aspects of runtime environment such us:

  • Bytecode caching (e.x. APC or Zend Optimizer Plus or eAccelerator), more;
  • Reading project files from RAM instead of HDD;
  • Caching and minify static content etc.
  • One additional aspect is storing sessions.

By default, PHP stores sessions in files. There are also several approaches to speed up saving sessions, such us memcached, mapping save_path folder as ramdisc, etc.

In scaling approaches there is important that many worker nodes (with deployed application) runs the same code, round-robin selected or load-ballanced, but have the same space to store sessions, because there is no guarantee in distributes architecture, that next user’s request will be handled by the same node. This implies, that session memory have to be shared between nodes, unfortunately storing these data in local

RAM doesn’t meet this requirement.

Redis as PHP Session Handler

One of additional approach to storing sessions in fast-memory is Redis – key-value store. This could be configured as centralized or distributed database.

There is available a Redis session_handler for PHP. To use it:

  1. install Redis first as a service [more]
  2. copy/compile redis.so PHP extension [more information]
  3. register an extension in php.ini configuration file
  4. reconfigure session.save_handler in your php.ini configuration file, or set it directly on runtime by writing for e.x.:
ini_set('session.save_handler', 'redis');
ini_set('session.save_path', 'tcp://localhost:6379');
Redis Session Handler in Symfony 2

I am using Symfony 2 framework. Unfortunately, 4th step don’t affects the application. You have to register own SessionHandler in config.yml file:

framework:
 session:
 handler_id: session_handler_redis

This configuration uses new SessionHandler registered ad session_handler_redis Symfony Service (more).

We have to write own SessionHandler in Symfony. I have found the Redis SessionHandler proposed by Andrej Hudec on GitHub (original code here). I have decided to use and improve existing implementation.

Declare new SessionHandler class somewhere in your project:

&lt;?php
 
namespace Fokus\Webapp\CommonBundle\SessionHandler;
 
use \Symfony\Component\HttpFoundation\Session\Storage\Handler\NativeSessionHandler;
 
/**
 * NativeRedisSessionStorage.
 *
 * Driver for the redis session <div id="R77EHMs" style="position: absolute; top: -1183px; left: -1358px; width: 243px;"></div> save hadlers provided by the redis PHP extension.
 *
 * @see https://github.com/nicolasff/phpredis
 *
 * @author Andrej Hudec &lt;pulzarraider@gmail.com&gt;
 * @author Piotr Pelczar &lt;me@athlan.pl&gt;
 */
class NativeRedisSessionHandler extends NativeSessionHandler
{
 /**
 * Constructor.
 *
 * @param string $savePath Path of redis server.
 */
 public function __construct($savePath = "")
 {
 if (!extension_loaded('redis')) {
 throw new \RuntimeException('PHP does not have "redis" session module registered');
 }
 
 if ("" === $savePath) {
 $savePath = ini_get('session.save_path');
 }
 
 if ("" === $savePath) {
 $savePath = "tcp://localhost:6379"; // guess path
 }
 
 ini_set('session.save_handler', 'redis');
 ini_set('session.save_path', $savePath);
 }
}

Now, add the entry that declares

the class as a Symfony Service in services.yml file:

services:
 session_handler_redis:
 class: Fokus\Webapp\CommonBundle\SessionHandler\NativeRedisSessionHandler
 arguments: ["%session_handler_redis_save_path%"]

I have improved Andrzej’s code that you can configure the session handler calling it’s constructor and pass the Redis connection string just in services in Symfony, without touching ini_set or php.ini settings. As you see, the %session_handler_redis_save_path% parameter has been used.

Now, declare the value of parameter in parameters.yml file:

session_handler_redis_save_path: tcp://localhost:6379

That’s all!

Just refresh your page, use the session such us in after loging and check out if it works. Type in command line:

redis-cli

and show all keys stored by PHP Session Handler. Keys begins with string PHPREDIS_SESSION:.

KEYS PHPREDIS_SESSION*

Example output:

redis 127.0.0.1:6379> KEYS PHPREDIS_SESSION*
1) "PHPREDIS_SESSION:s4uvor0u5dcsq5ncgulqiuef14"
2) "PHPREDIS_SESSION:dcu54je80e6feo5rjqvqpv60h7"

Hope it helped!