Our responses are faster when the response key location is compatible with target location than when it is incompatible, even when target location was irrelevant to the task (the Simon effect). This effect is observed even when two people shared the task (e.g., one pressing the left key and the other pressing the right key), known as the Social Simon Effect (SSE). The SSE has been attributed to the presence of the co-actor, which triggered automatic action co-representation (taking someone else’s action as their own). The present study examined whether the SSE can be observed with the presence of irrelevant objects in the visual scene (e.g., Japanese waving cat) but not a co-actor. Participants responded to the circle color in a 2-choice condition (e.g., a left key to red and a right key to green) or a go/no-go condition (e.g., a left key to red only). The irrelevant object was placed next to the monitor for half of the trials. We used both behavioral measures (response time and accuracy) and electrophysiological measures (brain activity) to assess response activation. For the 2-choice trials, we observed the Simon effect and lateralized readiness potentials, an index of response activation, in both object present and absent conditions. For the go/no-go trials, however, the effects were only obtained in the object-present condition. These findings suggest that the SSE is primarily caused by the spatial coding of one’s action relative to other events.