ConcurrentModificationException в forEach цикле

Имеется многопоточная среда обработки ивент-событий. Сам поток обработчик:

private class EventCallable implements Runnable {
    @Override
    public void run() {
        EventContainer event;
        while ((event = events.poll()) != null) {
            synchronized (event) {
                try {
                    if (event.future.isCancelled()) {
                        continue;
                    }
                    if (log.isDebugEnabled()) {
                        log.debug("Dispatching event {}", event.event);
                    }
                    event.future.running = true;
                    event.future.complete = false;
                    doDispatch(event.event);
                    event.future.set(event.event);
                } catch (Throwable t) {
                    event.future.setException(t);
                    log.warn("Exception in WorldEventDispatcher thread", t);
                }
            }
        }
    }
}

функция обработки:

private void doDispatch(WorldEvent event) {
    final ObjectID<?>[] objects = event.getDispatchableObjects();
    for (ObjectID<?> obj : objects) {
        if (obj == null) {
            continue;
        }
        if (!globalListeners.isEmpty()) {
            readLock.lock();
            try {
                final Set<WorldListener> notDispatched = CollectionFactory.newSet();
                for (WorldListener listener : globalListeners) {
                    if (!listener.dispatch(event)) {
                        notDispatched.add(listener);
                    }
                }
                notDispatched.forEach(globalListeners::remove);
            } finally {
                readLock.unlock();
            }
        }
        Set<WorldListener> worldListeners = getListeners(obj);
        if (!worldListeners.isEmpty()) {
            readLock.lock();
            try {
                final Set<WorldListener> notDispatched = CollectionFactory.newSet();
                for (WorldListener listener : worldListeners) {
                    if (!listener.dispatch(event)) {
                        notDispatched.add(listener);
                    }
                }
            } finally {
                readLock.unlock();
            }
        }
    }
}

Конструкция коллекции

private Set<WorldListener> globalListeners = CollectionFactory.newSet();
private ConcurrentHashMap<ObjectID<?>, Set<WorldListener>> listeners = CollectionFactory.newConcurrentHashMap();

функция вызова слушателя:

private Set<WorldListener> getListeners(ObjectID<?> id) {
    Preconditions.checkNotNull(id, "id");
    Set<WorldListener> set = listeners.get(id);
    writeLock.lock();
    try {
        if (set == null) {
            set = CollectionFactory.newSet();
            listeners.put(id, set);
        }
        return set;
    } finally {
        writeLock.unlock();
    }
}

Ексепшн летит при итерировании как listeners так и globalListeners коллекциях. Пробовал как синхронизацию так и как есть сейчас вариант на блокировках, итераторы. Собственно говоря, в чем может быть проблема? Сама ошибка :

[event-dispatcher-2] [22:07:25] [WARN] WorldEventDispatcherServiceImpl:     Exception in WorldEventDispatcher thread
java.util.ConcurrentModificationException: null
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) ~[?:1.8.0_60]
at java.util.HashMap$KeyIterator.next(HashMap.java:1453) ~[?:1.8.0_60]
at org.genfork.game.services.world.event.WorldEventDispatcherServiceImpl.doDispatch(WorldEventDispatcherServiceImpl.java:93) ~[classes/:?]
at org.genfork.game.services.world.event.WorldEventDispatcherServiceImpl.access$800(WorldEventDispatcherServiceImpl.java:34) ~[classes/:?]
at org.genfork.game.services.world.event.WorldEventDispatcherServiceImpl$EventCallable.run(WorldEventDispatcherServiceImpl.java:233) [classes/:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]

На строках - for (WorldListener listener : globalListeners) и for (WorldListener listener : listeners)