Communication is a key tool for facilitating multiagent coordination in cooperative and uncertain domains. We focus on a class of multiagent problems modeled as Decentralized Markov Decision Processes with Communication (DEC-MDP-COM) with local observability. The planning problem for computing the optimal communication strategy in such domains is often formulated with the assumption of the knowledge of optimal domain-level policy. Computing the optimal communication policy is NP-complete. There is a need, then, for heuristic solutions that trade-off performance with efficiency. We present a decision theoretic approach for computing optimal communication policies in stochastic environments which uses a branching future representation and evaluates only those decisions that an agent is likely to encounter. The communication strategy computed off-line is used in the more probable scenarios that the agent would face in future. Our approach also allows agents to compute communication policies at run-time in the unlikely event of the agents facing scenarios that were discarded while computing the off-line policy.